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ABSTRACT 

One of 15 core modules in a 22-module series desi^-od 
to train vocational education curriculum specialists (VECS) , thib 
guide is intended for use by both instructor and student in a variety 
of education environments, including independent study, team 
teaching, seminars, and workshops, as well as in more conventional 
classroom settings. The guide has five major sections. Part I, 
Organization and Administration, contains an overview and rationale, 
educational goals and performance objectives, recommended learning 
materials, and suggested reference materials. Part II, Content and 
Study Activities, contains the content outline arranged by goals. 
Study activities for each goal and its corresponding objectives 
follow each section of the content outline. Content focus is on the 
various purposes and components of decision-facilitation evaluations, 
the roles of the evaluator, the development or use of appropriate 
criteria and methodology for decision-facilitation evaluations, and 
the methods for preparing decision-facilitation evaluation plans and 
reports. Part III, Group and Classroom Activities, suggests classroom 
or group activities and discussions k* ed to specific content in the 
outline and to specitic materials in the list of references. Part IV, 
Student Self-Check, contains guestions directly related to the goals 
and objectives of the module, which may be urod as a pretest or 
posttest. Part V, Appendix, contains suggested responses to the study 
activities from part II and response- to the student self-checks. 
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PREFACE 



Who is a vocational education curriculum specialist? The answer 
to this question is not as simple as it might arnear. A vocational 
education curriculum specialist is likely to work in many different 
capacities, including, but not limited to: instructor, department 
chairperson, dean of vocational-technical education, vocational super- 
visor, principal, state or local director of vocational education, and 
curricul urn coordinator. 

The specialist is, perhaps, more identifiable by his/her respon- 
sibilities, which include, but are not limited to: 

• planning, organizing, actualizing, and controlling the work 
of an educational team performed to determine and achieve 
objectives . 

• planning, organizing, and evaluating content and learning 
processes into sequential activities that facilitate the 
achievement of objectives. 

• diagnosing present and projected training needs of business, 
industry, educational institutions, and the learner. 

• knowing, comparing, and analyzing different theories of curric- 
ulum development, management, and evaluation and adapting them 
for use in vocational -technical education. 

This teaching/learning module is part of a set of material s repre- 
senting a comprehensive curriculum development project dealing with the 
training of vocational education curriculum specialists. The purpose 
of this two-year project was 1) to design, develop, and evaluate an 
advanced-level training program, with necessary instructional materials 
based on identified vocational education curriculum specialist compe- 
tencies, and 2) to create an installation guide to assist instructors 
and administrators in the implementation process. 

The curriculum presented here is, above all else, designed for 
flexible installation. These materials are not meant to be used only 
in the manner of an ordinary textbook. The materials can be used 
effectively by both instructor and student in a variety of educa- 
tional environments, including independent study, team teaching, 
seminars, and workshops, as well as in more conventional classroom 
settings. 

Dr. James A. Dunn 
Principal Investigator and 
presently Director, 
Developmental Systems Group 
American Institutes for Research 
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PART I 

ORGANIZATION AND ADMINISTRATION 



Guidelines 

This study guide has five major sections. Each section contains useful 
information, suggestions, and/or activities that assist in the achievement 
of the competencies of a Vocational Education Curriculum Specialist. Each 
major section is briefly described below. 

PART I: ORGANIZATION AND ADMINISTRATION 

PART I contains an Overview and Rationale, Educational Goals and Performance 
Objectives, Recommended Learning Materials, and Suggested Reference 
Materials. This section will help the user answer the following questions: 

• How is the module organized? 

• What is the educational purpose of the module? 

t What specifically should the user learn from this module? 

• What are the specific competencies emphasized in this module? 

• What learning materials are necessary? 

• What related reference materials would be helpful? 

PART II: CONTENT AND STUDY ACTIVITIES 

Part II contains the content outline arranged by goals. The outline is a 
synthesis of information from many sources related to the major topics 
(goals and objectives) of the module. Study activities for each goal and 
its corresponding objectives foljow each section of the content outline, 
allowing students to complete the exercises related to Goal 1 before going 
on to Goal 2. 

PA RT III: GROUP AND CLASSROOM ACTIVITIES 

The "Activi ties-Resources" column in the content outline contains refer- 
ences to classroom or group activities and discussion questions related to 
specific content in the outline. These activities and discussion questions 
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are located in PART III and are for optional use of l ..her the instructor 
or the student. Both the classroom activitu snd J' cussion questions are 
accompanied by suggested responses for use as nolpful examples only--they 
do not represent conclusive answers to the problems and issues addressed. 
Also contained in the "Activities-Resources" column are the reference 
numbers of the resources used to develop the content outline. These 
reference numbers correspond to the numbers of the Suggested Reference 
Materials in PART I. 

PARL.IV: STUDENT SELF-CHECK 

PART IV contains questions directly related to the goals and objectives of 
the module. The self-check may be used as a pre-test or as a post-test, 
or as a periodic self-check for students in determining their own progress 
throughout the module. 

PART V: APPENDICES 

Appendix A contains responses to the Study Activities from PART II, and 
Appendix B contains responses to the Student Self-Check. The responses 
provide immediate feedback to the user and allow the module to be used 
more effectively for individualized study. They have been included in the 
last part of the module as appendices to facilitate their removal should 
the user wish to use them at a later time rather than concurrently with 
the rest of the module. 

Approximately 30 hours of out-of-class study will be necessary to complete 
thi s modul e . 

Overview and Rationale 

Evaluation is an integral part of the curriculum development process. 
Vocational education curriculum specialists must not only know the various 
types of evaluation, but must also be able to conduct or supervise evaluations 
and know how to use evaluation data in improving the curriculum. 
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The relatively recent call for accountability in education has required 
that teachers, students, administrators, the schools, and thr eurri'U'-un 
huutlf be evaluated. This module contains me^ !s and techniques for 
evaluating, or assessing, the effectiveness o< 1 -jurriculum. Evalu- 
ation should not be a one-time activity but rat. *n ongoing process 
that continually provides data to decision-makers either to help them 
improve the existing curriculum, or to ensure that a new curriculum 
is of the highest caliber. 

The first part of the module contains a description of the? activities 
performed during an evaluation for decision-rnaki.ig purposes. The pur- 
poses of each activity a- \s relationship to the other activities is 
explained. 

The second part of the module is a summary of the roles that ^valuators 
are required to assume in decision-facilitation evaluations. The third 
part is an identification of the criteria and methods used when uniduct- 
iru] evaluations. 

The last section contains exercises in putting all evaluation data 
together in a form tha f is useful to decision-makers. 
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Goals and Obje ctives 

ijpon ( cripli-t ifjn of t,hi', moduli-, f h* • -j-i#-ri r. will Do .ilih. 1 to achieve the 
* o I I ow i rnj '|o.j 1 s -ind oh M ,f 1. 1 v»". : 

f.OAl Is.l: -'SnW Hi! VAKIDU'- ("Jl'I'OSI S AND COMPONI NIS Dl DM J M ON- 1 AC 1 1 I - 
I A I ION i VAI HAT ION'.. 

hi, jf.i t i vf IS. 11 list the various pluses of a dec is ion- far i I - 
it.. it ion evaluation and st ite Mir» purpose 
I (>»• eac h phase . 

flliicr ! ivc lS.I^ I ist and »|ive e/amples ol tho evaluator 

activit if, performed in tho three stages of 
dec j s ion - f ao i 1 i til t. i on eva I ua t; ions . 

i )li I I i IS '.pcrity f.hi. ( hiit. ( forisfks of each typo of 
i /aluafion .if -/it./ and indicate how it. i'. 
rol.ji.cri to the other evaluation act i vi t ies . 

(,0AI IS.;': ANAIV/I AND INIM'I'KM INI I'OI I S I MA I I VAI NAHMS AHI Kl CJIJ1 Kl IJ 
10 ASSIIMI WHIN CONDUCIINO Dl MSI ON- 1 AM I I I A II ON I VAI t J A I IONS < 

ohjecMve !'><M Monti fy the ilul.ii>'. and functions of the 
persons responsible for conducting and/or' 
supervising the* various eva luat Ion activities. 

(;b)oMive IS,;V Solor.t the proper role of the evaluator in 
en h of tlii* dec Is Ion contexts. 

(,0a! ISM: 1)1 VI I OPOIMISI AIM'KOI'M AI I CM 1 1 MA AND Ml 11101)01 00Y 101' 
Dl MSlOfM AMI I IAI ION I VAI HAI IONS. 

Objective l''MI Identify the factors or<rlferia the evaliMtor 
must consider when detennln ln<| procedures 
for 1 collectim) dec I s ion- (Sic I I I U t Ion Infor- 
mation for #*#ti.h of the seversil ovaluat I nn 
iictivi t ies. 

Objective ] S , W Make informed decisions regard i ri «] the Mods 
r> f data tu MilliM I. and the f|**ru*r#i I methods 



to be used in collecting and analyzing 

that data for each of the several evaluation 

activities. 

Objective 15^3_3 Select appropriate instruments for collect- 
ing evaluation information. 

fiOAL 15.4: KNOW THE METHODS FOR PREPARING DECISION-FACILITATION EVAL- 
UATION PLANS AND REPORTS. 

Objective 15.41 Prepare an evaluation plan. 

Objective 15.42 Identify the proper times during the course 
of an evaluation when reports to the project 
or program operators should be submitted. 

Objective 15.43 Identify and prepare evaluation reports that 
will provide decision-makers with needed 
information about educational programs or 
projects . 
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Rec ommended Materials 

All nece^ary materials are included in this guide. 
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PART II 

CONTENT .*ND STUDY ACTIVITIES 



Goal 15.1 



Content Outline 



Activities-Resources 




Goal 15.1: Know the Various Purposes 
and Components of Decision 
Facilitation Evaluations. 




A. The CSE Model for Decision-Facilitation 
Evaluation 

1. The Center for the Study of Evaluation (CSE) 
Model for educational evaluation was chosen 
as the decision- facil itation evaluation model 
for use in this module because it is most 
consistent with the procedures and techniques 
advocated in Module 13, "Basic Concepts in 
Educational Evaluation" and Module 14, 
"General Methods and Techniques of Educational 
Evaluation." (1) (2) (26) 

B . ^ta^es^ of the CSE Model 



(1) "Methods and 
Theories of 
Evaluating 
Programs. " 

(2) Evaluation and 
Decision-Making 
The Title VII 
Experience is a 
particularly 
good source. 

(26) Educational 
Evaluation 
succinctly 
describes the 
CSE Model and 
its purposes, 
pp. 37-39. 



1. There are three major stages of decision- 
facilitation evaluation in the CSE model. 
Each stage has two phases and a purpose, or 
decision, toward which the evaluation is 
conducted: 

a. Preformative Stage 

(1) The first phase of this stage is needs] 
assessment. 

20 
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Content Outline (continued) 



(2) The second phase is program planning. 

(3) The purpose of this stage is to pro- 
vide information to the decision- 
maker so that goals may be selected 
and specific programs set up to meet 
particular needs. 

b. Formative Stage 

(1) The first phase in this stage is 
implementation evaluation. 

(2) The second phase is progress evalua- 
tion . 

(3) The purpose of this stage is to pro- 
vide information to the decision- 
maker on how the program is being 
implemented and operated and what 
results are occurring so that the 
program may be modified as needed. 

c. Summative Stage 

(1) The first phase in this stage is docu- 
mentation evaluation. 

(2) The second phase is outcome evaluation. 

(3) The purpose of this stage is to pro- 
vide information to the deci sion-rnaker 
on how the program was actually con- 
ducted and what the results were so 
that a decision can be made whether to 
continue or discontinue the program, 
or further modify it. 

In each evaluation phase, or evaluation 
activity, the evaluator must see that four 
processes are completed. 

a. Determine context (or decision area). What 
decision needs guide the evaluator's 
conduct? 

-10- 
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Content Outline (continued) 



b. Select appropriate information. 

c. Collect and analyze data. 

d. Report summary information.* 

All activities in the CSE model are inter- 
related. Figure 1 on page 16 of this guide 
shows the interrelationship of five of the 
evaluation activities. Figure 2 includes the 
latest addition to the model, the documentatior 
phase. * * 

Other decision-facilitation evaluation models 
may be used when the CSE model will not fit 
the situation; these are the CIPP model (30), 
and the Discrepancy model (28). 



See Discussion Question 
A in Part III. 



See Classroom Activity 
1 in Part III. 
'* See Discussion Question 
B in Part III. 

|(30) Educational Eval- 
uation and 



(28) 



Decision-Making . 

d iscrepancy 
Evaluation. 
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Study Activities 

Based on \jOnv v-!:id'rir; of the zunUmt en** line and the information pro- 
vided ului the iJtudj aerivitieo, com\>lete the fol lotting activities. 

Decisi o n-Facilitation Evaluation 

Evaluation for decision-making purposes is a continuous process. Ideal- 
ly the evaluator is involved with a program or curriculum from planning 
to development, through operation, and then on to the judgment stage. 
Because evaluation in vocational education is viewed as a continous pro- 
cess for providing information to decision-makers, the CSE model for 
evaluation (see Module 14) will be employed in this module. The CSE 
model was chosen not because its procedures are considered superior, 
but because they are very similar to those normally employed in the voca- 
tional curriculum development process. Most of the activities conducted 
when following the CSE model are also conducted in the CIPP model, the 
Discrepancy model, and other decision-facilitation evaluation models. 

The r '"F Model 

The CSE model defines educational evaluation as the process of determin- 
ing the kinds of decisions that have to be made; selecting, collecting, 
and analyzing the information needed in making those decisions; and then 
reporting the information to the appropriate decision-makers. Evaluation 
information is gathered and reported by the evaluator to help decision- 
makers decide among optional courses of action when faced with problems 
such as how a program might be improved, or whether a program should be 
modified, continued, or terminated. 

There are three major stages to the CSE model, the Preformative, the 
Formative, and the Summative. In each stage two major evaluation activi- 
ties are conducted for specific decision purposes. 
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In the Preformative stage are the evaluation phases, or activities, 
of needs assessment and program planning . The purpose of needs assess- 
ment (which is always performed when developing vocational curricula) 
is to provide information for goal selection. The major purpose of 
program planning evaluation is to provide information that will aid in 
the selection or development of a program. In essence, needs assess- 
ment asks what will be needed, program planning evaluation asks what 
program will meet those needs. 

The Formative stage of the CSE model is concerned with what is being 
done and what is resulting. The purpose of both phases, or activities, 
of the Formative 'jo, Implementation and progress evaluation , is to 
provide information that will help the decision-maker modify the pro- 
gram while it is still flexible. 

The Summmative stage of the CSE model is concerned with what the end 
results were. The documentation evaluation phase, or activity, in- 
volves collecting information that shows how the program was actually 
implemented; the outcome evaluation measures the results or outcomes 
obtained by the program. The purpose of both activities of the Summa- 
tive stage is to provide information to the decision-makers that will 
help them decide if a program should be continued, discontinued, or 
further modified. 

1. List the three major stages of the CSE model for educational 

evaluation, the major purpose of each stage, and the decision area 
serviced by each stage. 
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CURRENT MODEL 

i)ffimtii)i\. TIic current model defines ediiout/onrd aval- 
f jon as the process of defonnmm.n lUu kinds uf deci- 
sions t\\(it hove to be made; se/nctimj, f:oll(!(:Ii/!|;, unci mui- 
Jy/int» in f r irninh'on needed in making these (Incisions; and 
thru reporting this informr/Moii lo uppvapriaU) decision 
nuiUtifa. Thus, evaluation information should help deci- 
sion in.ikcrs in dr- anions alternative courses of 
action such a : Mow a p njjrnm mi^ht he unproved. As 
may he seen ' an the figure on tin; next pa^e, the Center 
has identilied mr major r mils of decisions that have to 
Im! made ( jnr I i< I ,1 hy diamonds) md these are associated 
with five phase* of evaluation activities. Tin; hasic fea- 
tures of these decisions and phases .in* as follows: 

Weds Assessment involves statin" potential educational 
tjoals or objectives (preferably in terms of student per 
foimanoe rather than instructional processes), deciding 
•aIucIi of thi'se .ire ol highest priority, and determine^: 
how well the existing educational proijram e- n oetfnfi 
these objectives. This latter information is then used e. 
the decision maker to identify the major needs so th.it h" 
can decide which ones should be attacked. A school 
superintendent, for (!xam|)le, ini«ht have a needs assess- 
ment conducted in his district to help him decide where 
educational programs should be developed or improved 
For instance, it mi^ht be found that the students at one 
school an? not dniuj> us well as they should in chemistry 
while at another school the major deficiencies alight he 
in foreign lanjniaejis. It mi«ht also be disclosed that im- 
provements are needed in student performance in Kn»lish 
throughout tin? district. Thus, needs assessment f indiums 
are used in determiiiiiiM which problem areas Miould be 
attacked. 



Program Plwmini* involves making decisions about the 
kinds of programs or combinations of programs (or program 
components) that should be adopted to meed tin.* problems 
identified in tin; needs assessment. Thus, a series of 
decisions are made about how the needs miuht best be 
met with the resources available to do the job. This ac- 
tivity usually involves a series of planning meetings that 
should result in a written document describing how tin; 
school or project intends to achieve the desired objec- 
tives. During the program planning phase, the (.'valuator 
siiKKcsts techniques to facilitate; planniiiH decisions, pro- 
« ides advice re^ardin^ evaluation requirements for alter- 
native plans, and builds into the final plan the procedures 
necessary for carrying out subsecpnmt evaluation activi- 
ties. 

/mplcmou/u/io/i Kvu/uofion focuses on whether tin; pro- 
cedures specified in tin; program plan are actually carried 
out in the intended manner. Thus, it involves investigat- 
ing the decree to which the program plan has been 
adopted properly to the field situation. Typical implemen- 
tation questions for which tivaluation information is 
need* d mi^ht be "Did the hooks arrive on time?" and 
"Are the students enrolled in the program the ones for 
whom it was intended?" 
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Progress FA'uhiution, on the other hand, is aimed at de- 
termining tin? extenl to which the program is actually 
making gains towards achieving its objectives. Since a 
program may be implemented exactly as planned but still 
no! reach its intended objectives, it is necessary to inves- 
tigate whether the plan is really a good one to achieve the 
student needs. Further, it is obviously wasteful to install 
a program in a school in the Fall and then wait until 
Spring to learn that it failed or that it might have been 
improved if corrective action had been taken earlier- 
It is apparent, therefore, that decision makers need infor- 
mation ab 1 m ' nient progress during the course of a 
program so 1 if problems develop they can be identi- 
fied and corp'i.ied quickly. 

At this point, it is important to note certain similarities 
.mil differences between implementation and progress 
evaluations. Both kinds of activities fall under the head- 
ings of "process" or "formative" evaluations and deal with 
the extent to which the program is functioning properly. 
Further, both may lead to decisions regarding possible 
changes anil mollifications in how the program is being 
run. Implementation evaluations, however, deal with the 
extent to which the program's procedures are implemented 
as planned, whereas progress evaluations are aimed at 
determining the extent to which these procedures are 
producing the desired gains in student performance. De- 
cisions about modifying the program will, of course, rely 
on both kinds of data since there may be problems in 
how the program plan is behg implemented as well as in 
tin plan itself, 
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i )uN nnif l-Vu/ihifiuiis lead In final jii(l«in«Mits regarding 
;nr general worth t > t a total program [as opposed to prog- 
ress evaluations that ileal mainly with program compo- 
ufMits and are dour continuously throughout the program's 
life). Thus, outcome evaluation information is used in 
making decisions such as "Should we continue the pro- 
gram next year?" and "Should we extend the program to 
other schools in the district?" 

Heeycling hoops. The Center's model presents the five 
kinds of evaluation activities in a logical sequence cor- 
responding to the usual development and operation of an 
educational program. It is apparent, however, that some 
of these activities, especially implementation and prog- 
ress evaluations, may he overlapping in time. It is also 
apparent that decisions made at one point in a program 
may require repeating one or more of the preceding 
phases. A progress evaluation, for example, might indi- 
cate poor student performance on certain ohjectives. A 
special needs assessment might then disclose that the stu- 
dents did not have the requisite reading skills for the in- 
structional materials specified in the program plan that was 
adopted to help them achieve these objectives and, there- 
fort*, additional planning is needed. To avoid clutter, all 
these recycling loops have been deleted from the figure, 
but .ire implied by the dotted arrows stemming from each 
of the major decisions. If all the recyclir , and feedback 
loops that might potentially be necessary were included 
in the figure, there- would essentially be a line from each 
box to every other one. 



Mode/ Consistency. One important leature of the model 
is that it has certain consistencies across the five phase^j 
For example, each phase starts with a context determin™ 
lion. The popose of this activity in needs assessment is 
primarily to determine t ho scope and level of the evalua- 
tion (e.g., j.re we evaluating a school or a particular read- 
ing program in that school?). Context determination also 
includes an investigation of the resources, constraints, 
social dynamics, political pressures, personalities, and 
environmental conditions that might influence decisions 
about how program and evaluation activities should be 
conducted. The nature and focus of context determina- 
tions do, of course, change from phase to phase. In pro- 
gram planning, for example, the evaluator would take 
into consideration the personality characteristics and 
biases of the planners whereas in implementation evalu- 
ation he may focus on potential environmental constraints 
that may inhibit the program from being run as planned. 

Another important consistency is that the second step 
in each phase involves "setting up" the procedures that 
will be used in that phase. In other words, a written plan 
should be developed along with an explication of the 
rationale for it. This plan describes how the activities in 
that phase of the model will be conducted. As noted 
above, it is always possible to revise plans and proce- 
dures through recycling, but it is usually better to start 
with a clear idea of what you intend to do than to as- 
sume that so many changes will occur as to make it not 
worth the effort. a 

Finally, it is important to note that all evaluation activ^ 
ties in each phase eventually lead to a report to the deci- 
sion maker who in turn determines whether to drop the 
project at that point, recycle, or go on to the next phase. 
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Documentation Phase 

Since the previous explanation was written, another phase has been 
added to the CSE model. That phase, the documentation phase, and 
what was previously the outcome stage, now comprise the Surmative stage. 
(See the material for Activity 1 and Figure 2.) Evaluator activities 
in the documentation phase consist of such tasks as collecting informa- 
tion on how the program was actually implemented and putting that 
information into a form that the decision-maker can use. The evalua- 
tion process, according to the CSE model then, consists of three stages, 
each of which has two distinct evaluator phases or activities. Each 
phase includes the processes of (1) determining the decision area of 
concern; (2) selecting appropriate information, (3) collecting and 
analyzing data; and (4) reporting summary information to decision- 
makers. Specific examples of evaluation activities that might be 
performed in the CSE from vocational education will be included in the 
following readings. 
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FIGURE 2 
AN EVALUATION MODEL 
(From the Center for the Study of Evaluation, UCLA) 
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2. List the six evaluation phases, or activities included in the CSE 
model for educational evaluation. List also the four processes that 
make up each of the six phases and give an example (from your 
experience) of an activity that an evaluator for a vocational 
education program might perform during each of the four processes 
in one phase of the CSE model. 

Evaluation Activity Characteristics 

Needs Assessment . The primary focus of the needs assessment phase of 
the CSE model is on identifying and delineating the goals for a project 
or program. There are three distinct characteristics of a needs assess- 
ment for vocational curriculum development, and hence, three distinct 
activities in which the vocational education evaluator will be involved. 
These three activities are population needs assessment , job market analy- 
sis , and occupational performance requirements analysis (task analysis). 



Population Needs Assessment . The population needs assessment is a tech- 
nique or process for establishing an information file that describes the 
population being served. To be complete, the population needs assess- 
ment file must contain the data elements required to establish the rela- 
tive vocational education needs of the target population. The population 
needs assessment, and subsequent analysis, can show how relevant the ex- 
isting goals and objectives of the vocational education system are to the 
social, cultural, and economic problems of the target population. Needs 
assessment data can serve as a basis for new objectives and goals; it 
can also serve as the basis for determining the success of existing vo- 
cational curriculum goals. An accurate, complete needs assessment is one 
of the foundation blocks for the vocational curriculum. All subsequent 
phases of the curriculum development and evaluation processes are based 
on it. 
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In some cases the population needs assessment is conducted by the cur- 
riculum specialist alone; in other instances, the evaluator will con- 
duct it. The evaluator's primary role in this function is to develop 
or help develop potential educational goals and objectives, establish 
priorities among those objectives, and then determine how well existing 
curricula or programs meet them. 

Job Market Analysis . Job market analysis is the labor supply-labor de- 
mand counterpart to the population needs assessment. While population 
needs assessment is a technique for estaolishing an information file on 
the needs of the target population, job market analysis is a method for 
establishing an information file of the needs of the occupations that 
fall within the categories served by vocational education. The evalua- 
tor's role in job market analysis is to (1) identify and determine the 
validity and effectiveness of the criteria used in conducting the analy- 
sis, and (2) ensure that the labor market analysis data are closely cor- 
related with population needs assessment data and job performance re- 
quirements analysis. Close coordination of the three types of data is 
important to assure that all are used in curriculum development and pro- 
gram planning. As with population needs assessment, the evaluator helps 
the program planner establish educational goals and objectives and es- 
tablish priorities among them based on the gathered data. 

The emphasis of job market analysis is on determining the areas in which 
jobs are developing or declining, and then identifying the factors that 
contribute to this expansion or decline. Many sources are used in estab 
lishing the job market analysis file, including employment development 
services, Chambers of Commerce, labor organizations, and advisory commit 
tees. The most common data gathering device for job market analysis, 
however, is the questionnaire. The evaluator may play a key role in de- 
veloping and validating the questionnaire and then assist in analyzing 
the returned data. 
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Occupationa l Performance Requirements Analysis (Task Analysis). Once 
population needs assessment and job market analysis have been completed 
and tentative educational goals and objectives have been formulated, job 
specifications for the vocational instructional program must be estab- 
lished. Such specifications include the identification of the skills 
and knowledge required to complete or perform a given job, and usually 
include information on the frequency with which a given task is performed 
and its relative degree of difficulty. A task analysis is usually per- 
formed with the help of individuals having skills and substantial know- 
ledge in the occupational field for which instruction will be given. 
Questionnaires and interviews are two of the most common techniques for 
conducting task analyses, and the evaluator's role may include develop- 
ing and validating the questionnaires and interview schedules. 

When the population needs assessment, the job market analysis, and the 
occupational performance requirements analysis have been completed, the 
evaluator's role is to help coordinate and correlate all data and help 
develop the educational goals and objectives. This coordinated effort 
is considered the needs assessment stage of both the curriculum develop- 
ment process and the decision-facilitation evaluation process. 

Program Planning . The program planning evaluation phase, which is also 
part of the preformative stage, is an outgrowth of the needs assessment 
phase. Its major objective is the integration of information obtained 
in needs assessment and information such as budget constraints and area 
labor plans, for the purpose of determining the kinds of curricula that 
will meet all needs. During this phase, the evaluator (1) suggests and 
helps develop systems' or techniques to assure that all needs and objec- 
tives are included, (2) assures that supportive services for the pro- 
posed program are adequate, (3) identifies deficiencies that might in- 
hibit development of the program, (4) determines whether the scope and 
sequence of the courses or programs are flexible enough and attainable 
by the students, and (5) ensures that techniques and methods for facili- 
tating final evaluations are included in the overall plan. 
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I mp! ementation Evaluation and Progress Evaluation (18). Implementation 
evaluation focuses on whether or not the procedures specified in the 
program plan are actually carried out in the intended manner. It in- 
volves investigating the degree to which the program plan has been adapt- 
ed properly to the field situation. Typical implementation questions 
for which evaluation information is needed are: Did the books arrive 
on time? and Are the students enrolled in the program the ones for whom, 
it was intended? 

Progress evaluation, on the other hand, is aimed at determining the 
extent to which the program is actually achieving its objectives. 
Since a program may be implemented exactly as planned but still not 
reach its intended objectives, it is necessary to investigate whether 
or not the plan is really a good one for achieving student needs. Fur- 
ther, it is obviously wasteful to install a program in a school at the 
start of the fall semester and then wait until spring to learn that it 
failed or that it might have been improved if corrective action had 
been taken earlier. Decision-makers need information about student pro- 
gress during the course of a program so that if problems develop, they 
can be identified and corrected quickly. 

It is important to note certain similarities and differences between 
implementation and progress evaluations. Both kinds of activities fall 
under the heading of process or formative evaluations and both are 
concerned with the extent to which the program is functioning properly. 
Further, both may lead to changes and modifications in a program. Im- 
plementation evaluations, however, are concerned with the extent to 
which the program's procedures are implemented as planned, whereas 
progress evaluations are aimed at determining the extent to which these 
procedures are producing the desired gains in student performance. De- 
cisions to modify the program will, of course, rely on both kinds of 
data since there may be problems in both areas. 
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Documenta tion Evaluation 

— - — j 

The documentation evaluation phase is part of the summative stage, and 
involves collecting information that shows how the program was actually 
implemented. The key word in this phase is "implemented." The evalua- 
tor determines how the program was actually put into operation and how 
it may have differed from what was planned. Evaluator activities might 
include such things as observation and comparison with the program plan 
and interviews with students, teachers, and program planners. The in- 
formation gathered and reported to the decision-maker is essential in 
interpreting the results of the other summative phase—outcome evaluation. 

Outcome Evaluation 

The final phase in the CSE evaluation model is the outcome evaluation, 
a process that provides information to decision-makers that will help 
them decide whether to extend, continue, revise, or terminate a program 
or course. The outcome evaluator measures the results or outcomes ob- 
tained by the program, such as placement rates, job success, student 
knowledge and ability, and employer and student satisfaction. The eval- 
uator's carefully assessed measures are reported to the decision-maker, 
usually with recommendations for action, for the decision-maker's final 
action. 



3. State the primary characteristics (or purposes) of decision- 
facilitation evaluation at each of the phases, and relate the 
evaluation results that are. obtained to the next phase of the 
decision-facilitation model. Your answer may be written in 
narrative form but be specific. Begin with the purposes and 
characteristics of needs assessment evaluation and proceed to 
the successive phases. 

(See Appendix A for possible answers.) 
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Goal 15.2 



Content Outline 


Activities-Resources 








77777777777777/////////^ 

//X%Gcdl 15.2: Analyze and Interpret the ■////// 
////, Roles That Evaluators Are Required to Z^V/ 
/7%^Assume When Conducting Decision- //////' 
•//7%Facil itation Evaluations. vZov/ 


A. Primary Evaluator Purpose in the CSE Model 

The primary purpose of the evaluator in the CSE 



evaluation model is to assist the decision-maker in 
reaching conclusions or decisions about a program. 
To do this, the evaluator must assume a number of 
roles during the program. 

B. Evaluator Roles in the C SE Model, 

The most common evaluator roles in each phase are: 

1 . Needs assessment: 

a. determining the need(s) for a program 
(context) ; 

b. assisting in selecting objectives; 

c. assisting in rating the importance of 
objectives; 

d. determining if other programs meet any of 
the objectives; 

e. determining the importance of the needs 
identified; and 

f. reporting the results of the assessment. 

2. Program planning: 

a. determining the context in which the pro- 
gram planning will take place (What has to 
be done?); 
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Content Outline (continued) 



b. assisting in program selection or develop- 
ment; 

c. assessing quality and quantity of support 
services ; 

d. identifying deficiencies that might hinder 
program operation; 

e. determining if objectives can actually be 
met with this program; 

f. building evaluation into the program; and 



a. describing how the program is being 
operated; 

b. comparing the operation with the plan; 

c. comparing the plan with other plans or 
programs; 

d. determining if planned procedures a?e 
leading to specified outcomes; and 

e. reporting the results of the implementation 
evaluation. 

4. Progress evaluation: 

a. determining and describing progress; 

b. determining if progress is satisfactory; 

c. comparing progress against a norm of some 
kind, such as another program or predeter- 
mined criteria; 

d. determining the readiness of students to 
progress; and 

e. reporting the results of this evaluation. 



g. reporting the results.* 
3. Implementation evaluation: 



* See Classroom 
Activity 2 in 
Part III. 
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Content Outline (continued) 



Documentation evaluation: 

a. describing how the program was actually 
conducted; 

b. comparing the operation with the pi : r id 

c. reporting the results. 
Outcome evaluation: 

a. determining the results of the program; anc 

b. reporting the outcome of the program.* * See Discussion 

Question C in 
Part III. 
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Study Activities 

Hased on your reading of the content outline and the information pro- 
vided with the study activities j complete the exercises that follow. 

.. LUATOR RESPONSIBILITI ES 

he duties and functions of the evaluator are listed according to the 
pnase of the CSE model for educational evaluation in which they occur. 
Within each phase, these duties are given in the order in which they 
would normally occur. 

Needs Assessment 

1. Determine or assist the curriculum planner to determine and list the 
full range of potential educational objectives for the course or pro- 
gram being planned. Sources of objectives for vocational education 
courses or programs include potential students, the community at 
large, the occupations or businesses to be served, and the curri- 
culum planners themselves. 

2. Help the decis^ >n-maker place a value on the objectives by determin- 
ing their relative importance to the communities that vocational 
education serves (i.e., the students and business and industry). 

3. Determine the degree to which the adopted objectives are being 
met by other existing programs. 

4. Uie the information obtained in activities 1, 2, and 3 to help the 
decision-maker determine the final relative importance of the needs 
that were defined. The decision-maker will then decide which ob- 
jectives to adopt for the course or program. 

?T.93T^}\ Planning 

1. Determine the context In which program planning will take place. 
This usually involves a series of planning meetings that result 
in a document describing how the decision-makers Intend to achieve 
the desired objectives. 
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2. Suggest and help develop systems or techniques to ensure that all 
needs and objectives are included. This activity may include 
helping planners develop and implement procedures for selecting 
among optional programs. 

3. Assure that supportive services are adequate for the proposed course 
or program. 

4. Identify any deficiencies (such as staff attitude and finance) that 
might inhibit development of the program. 

5. Determine whether the scope and sequence of the courses or programs 
are flexible and the objectives attainable by the students. 

6. Ensure that techniques and methods of facilitating final evaluations 
are included in the total program plan. The evaluator may place 
certain restrictions on the conduct of the program, such as deter- 
mining whether (or which) tests are appropriate, and determining 
whether students should be randomly assigned to it. These restric- 
tions are imposed in order to ensure that the evaluation information 

* system is built into the program plan. 

Implementation Evaluations 

1. Describe how the program is being conducted so that others may rep- 
licate (or avoid) its procedures. 

2. Determine if a program is being conducted as planned. This function 
may. prove useful when attempting to explain the reasons for progress 
or outcome evaluation results. 

3. If possible, compare what has happened so far in the program with 
what happened in different programs. This function will help future 
program planners select effective procedures and avoid ineffective 
or poor ones. Comparison and documentation will help the present 
program planners or operators anticipate problems before specific 
procedures are incorporated into program plans. 

4. Determine if program procedures are leading to the specified, 

^ planned behavior. This function is conducted in order to provide 
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information needed to modify planned procedures or to get planned { 
procedures implemented properly. 

Progress Evaluations 

1. Describe the progress at each point in the program in order to have 
a basis for comparison if the program is replicated. These data 
will serve as benchmarks for future program operators. 

2. Determine the relative degree of satisfactory progress. This infor- 
mation may be required fo>- roports to program monitors or funding 
agencies . 

3. Compare the progress made to a standard or other program in order 
to identify the most effective one. This information will help 
decision-makers weed out less effective programs. 

4. Determine exactly what progress is being made in the program. This 
information is needed in order to identify reasons for low or high 
level performance so that the program can be modified. 

5. Determine whether individual students or groups are ready for the 
next phase of the program. This function also helps the decision- 
maker modify the program as needed. 

Doc u mentation Eva lu ations 

1. Collect and document information that shows how the program was 
actually conducted. 

2. Compare the way the program was actually conducted to the way it 
was planned. 

Outcome Eval uations 

1. Measure the results or outcomes obtained by the program. For voca- 
tional education, these results might include placement rates, 
student knowledge and proficiency in performing the activities or 
jobs that they were taught,* job success in terms of promotion 
and self-satisfaction, and employer satisfaction with the students, 
their attitudes and abilities. 
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Report the results of the program to the decision-maker. The 
evaluator usually includes recommendations for extension, revision, 
or termination of the course or program. 

For each of the following descriptions of evaluator responsibilities, 
indicate the evaluation phase at which the activity is conducted. 

a. Selecting a set of objectives for a course from a list of 
instructional objecti ves . 

b. Reviewing the results of the program in order to decide whether 
that program should be continued or dropped. 

c. Developing procedures to investigate which of several kinds of 
instructional materials should be used in the program. 

d. Establishing standards of performance for a program that might 
be installed to determine whether students are exceeding, 
achieving, or failing to reach proficiency on objectives. 

e. Preparing an interim report on whether or not the program is 
on schedule in meeting its instructional objectives. 

f. Deciding upon the appropriate evaluation design to ascertain 
the effectiveness of the program. 

g. Investigating whether students in a program are using any 
special equipment properly. 

h. Determining whether the procedures that were planned for the 
operation of the program were in fact followed. 

i. Observing teachers to determine what they might be doing to 
account for their classes 1 unusually high or low test scores. 

j. Preparing a summary report detailing which students appeared 
to profit most from the program. 

k. Developing effective methods of combining the ratings of objec- 
tives which have been rated by a variety of individuals such as 
parents, administrators, students, workers, and employers. 

1. Interpreting i:he students 1 scores on the weekly evaluations 
for the teacher's own records. 
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Going back over the data collected during the course of the 
program to find out whether student interest in the subject 
matter increased, decreased, or remained the same. 
Using a checklist to determine if the teachers followed the 
specially prepared lesson plans detailing the sequence for a 
given unit of instruction. 

Reporting the students' average score on each of the objectives 
of the program's first unit. 

Deciding how the data about a program will be analyzed and 
reported. 
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Content Outline 



Activities-Resources 
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Goal 15.3: Develop or Use Appropriate 
Criteria and Methodology for Decision 
Facilitation Evaluations. 




Evaluation Criteria and Methodology 

1. The determination of evaluation criteria is 

a function of the first activity performed in 
each phase of decision-facilitation evalua- 
tion, that is, determining the context. If 
the context within which the evaluation will 
take place is known, the identification of 
evaluation criteria follows naturally. 

2. To determine the context, the evaluator asks a 
series of questions relative to the activity 
being perfcrmed—that is, relative to the 
phase--in the curriculum development process, 
a. Needs assessment - Since the purpose of 

this phase is to determine what kind of 
program is needed, the evaluator works* 
closely with the curriculum developers, 
and in some instances may supply them with 
data in addition to validating the needs 
they discover. The primary activities of 
the evaluator in the needs assessment 
phase are: identifying or validating the 
needs of the community, the students, and 
business/industry for vocational educa- 
tion; analyzing the roles of the workers 
in the occupations selected; and selecting 
objectives for the proposed program (22). 



(22) Evaluative 
Cri teria. 
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Program planning - Evaluative criteria for 
this phase, in the form of items or data 
that may be obtained, are rather difficult 
to discern. The evaluator's role is usu- 
ally to assist the program planners in 
ensuring that a logical planning process 
is followed, based upon and directed 
toward the needs identified in the previ- 
ous phase. The evaluator helps the 
decision-makers integrate program needs 
through the use of appropriate techniques 
and instructional methodology, and over- 
sees the planning to ensure that all 
factors pertinent to the successful opera- 
tion of the program are Included. * 
Implementation evaluation - Being part of 
the formative stage, one purpose of this 
evaluation activ»'y is to provide informa- 
tion to the decision-makers relative to 
modifying the program before it becomes 
completely operational. The CSE model 
contains provisior for conducting imple- 
mentation evaluations before and after 
instruction begins a »d at the end of a 
major irit or comi ,i<rnt of the program. 
In addition. ' r h< 'aluator in this phase 
documents tno extent to which the program 
is being implemented as planned, and notes 
any discrepancies between operation and 
plan. * * 

Progress evaluation - Being part of the 
formative stage, criteria examined here 
should also relate to modifying the pro- 



* See Discussion 
Question D in 
Part III. 



See Discussion 
Question E in 
Part III- 
See Discussion 
Question F in 
Part III. 
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gram in order to improve it. The evalua- 
tor asks: What results are being obtained 
by the program? and then compares these 
results with other programs if possible 
and with stated objectives.* 
Documentation evaluation - This activity, 
conducted when the program is complete, 
is primarily summative evaluation although 
project records collected during the life 
of the program must be saved. The primary 
activity is to document the actual opera- 
tion of the program both to help in inter- 
preting outcome data and to provide infor- 
mation for future program planners.* 
Outcome evaluation - In this phase, cri- 
teria are examined in order to place a 
value on the program. What actually 
happened as a result of the program? 
What was the cost of the program? How 
does that cost compare with similar 
programs? 



See Discussion 
Question G in 
Part III. 



See Discussion 
Question H in 
Port III. 



B . Data Identification and Col lection 

Once criteria for evaluation have been determined, 
the evaluator must first identify the data that 
ran be collected in order to examine those crite- 
ria, and then collect and analyze the appropriate 
data. 

1. Data identification - If the objectives of a 
program have been clearly stated in observable 
or measurable terms, the identification of 
necessary data follows relatively easily. If 
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objectives are stated for the program planners 
and operators as well as for the students or 
participants, even process information, such 
as that needed for needs assessment, implemen- 
tation, and documentation evaluations is 
easier to obtain (4) . * 

Data collection - The three largest variables 
in the data collection process are the data 
sources, the methods used to obtain data from 
the sources, and the people or devices used to 
collect the data (3). Data sources may in- 
clude individuals in the program such as stu- 
dents or participants, those involved in plan- 
ning and operating the program, and others in 
a position to observe or be affected by the 
program. 

Various methods are available for oDtaining 
evaluative data from these sources. The most 
common meth/is used 'in education are tests of 
various types, observations, and records. 

The people or devices used to collect data 
must be consistent and reliable so that the 
data they collect is also consistent and reli- 
able and can be used in decision-makinq. 
Three types of indexes are typically used to 
measure the outcomes of processes cr programs. 
These indexes are: 

a. behavioral, which measure the changes or 
modifications in behavior that occur as a 
result of a program; 
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* See Discussion 
Question I in 
Part III. 
(4) Taxonomy of Edu- 
cational Objec- 
tives , Handbook 
1_: Cognitive 
Domain . See also 
(5) "The Indirect 
Assessment of 
Social Attitudes." 
See also: (7) 
"How to Write 
True-False Items." 
See Also: (8) 
Measuring E duca- 
tional Achievement . 
See Also"! UT) 
Constructing 
Achievement Tests 
See Also 



(3) 



(T9j 

Supervision : A 
Synthesi s of 
Thought and Actio n. 
See also: (20T~ 
Preparing Instruc- 
tional Objectives . 
See also: (21) 
Multiple Choice 
Questions : A Close 
Look . See Also: 
(23) Measurement 
and Eval uation in 
the Classroom . See 
al sol (27) An Eval- 
uatio n Gui deboo k. 
See uTsol (3D 
Unobi "si ve Measures : 
NonreaVtTv e Research 
i the" Social 
Scien ce!^ See also: 
(32) Tes>. Cc^ structi o 
Eval '>g s e *JTC j^ 
Str^ gi c~ :V? 
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b. other outcomes, which measure such things 
as changed values and attitudes; and 

c. process, which describe the program rather 
than measure outcomes. Process measures 
may be used as devices for interpreting 
the results of behavioral and other out- 
come measures. 

Evaluation designs— The most common evaluation 
designs are those discussed by Campbell and 
Stanley (6), and by Popham (27). Major ideas 
regarding this general quasi-experimental clas 
of designs are: 

a. Statistical design is not synonymous with 
evaluation. (Statistical analysis is one 
of the tools the evaluator uses to arrive 
at decisions about program effect.) 

b. The purpose of statistical design is to 
organize and analyze information about 
program outcomes in such a way that clear 
evidence of the program effects is re- - 
vea'led. (Statistical analysis might in- 
include a follow-up cf students to deter- 
mine percentage of graduates in each of 
the following categories: 

(1) in a job directly related to the 
training; 

(2) in a job with some relation to 
training; 

(3) in armed services with assignment 
related to training; 

(4) in armed services not related to 
training; 
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and Quasi- 
Experimental 
Designs for 
Research . 

[27) An Evaluation 
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(5) continuing in school for more 
training; 

(6) continuing in school but not in a 
related major; 

(7) unemployed and not looking for a job 
in the field of his training.) 

c. Program effect is demonstrated by compar- 
ing outcome measures for students enrolled 
in the program under consideration with 
outcomes for similar kinds of students not 
enrolled in the program. Evidence of pro- 
gram effect can also be based upon compar- 
ison of program student outcomes and out- 
comes for an appropriate norm group, or 
with outcomes for students enrolled in 
similar programs prior to the introduction 
of the program being evaluated. (A com- 
parison through a follow-up of graduates 
from vocational education and from general 
education as to progress on the job.) 

d. Program effects should be both educational- 
ly and statistically significant (raising 
job placement from 54% to 56°/ is trivial). 

e. Interpretation of the significance of the 
difference in outcome indicators for pro- 
gram and comparison students must be reli- 
able and valid. Validity in general means 
that program objectives rather than other 
factors are being measured; reliability in 
general means that the objectives are 
measured 'wi th precision. (Follow-up sur- 
veys based on mail returns in which the 
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response level is low--less than 50%— are 
not valid or reliable unless there is an 
attempt to spot check those who did not 
respond by using telephones and/or person- 
al contact to determine if the mail data 
response is in agreement.) 
Data analysis - Most data can be analyzed 
using descriptive statistics. (An important 
consideration is that the data be analyzed 
using the smallest independent units avail- 
able, whether that be individual students, 
classes, or schools.) (6) 

a. The use of descriptive statistics for edu- 
cational evaluation differs from that for 
educational research. In educational re- 
search, the statistical results are used 
to confirm or reject hypotheses; in educa- 
tional evaluation, statistical results are 
used to indicate differences achieved as a 
result of a given treatment. Decisions 
are then made on the basis of the size or 
magnitude of the differences indicated. 

b. The analysis of evaluation data also pro- 
vides information on cost/effectiveness. 
The evaluator provides this information so 
the decision-maker can decide if the pro- 
gram effects were worth the program costs. 
Cost/effectiveness comparisons of two or 
more programs and instructional delivery 
systems are often conducted.* 
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(6) Experimental 
and Quasi- 
Experimental 
Designs for 
Research . See 
also: (7) "How 
to Write True- 
False Items." 
See also: (8) 
Measuring E duca- 
tional Achieve- 
ment . See also: 
(12) Constructing 
Achievement Tests, 
See also: (23l 
Measurement and 
Evaluation in 
the CI assroom . 
See aTscK (27) 
An Evaluation 
Guidebook , 
pp. 234-257. 



See Classroom 
3 in Part III. 



Study Activities 

Based on your reading of the content outline and the inform if ion pro- 
vided with the study activities, complete the activities :> i: fallow. 

EVALUATIVE CRITERIA 

There is a series of questions that educational evaluators must ask 
themselves, the decision-makers they serve, or the program operators, in 
order to determine procedures or techniques for collecting decision- 
facilitation information. The following questions are examples of the 
types of criteria about which the evaluator needs information during each 
of the evaluative phases. 

Needs Assessment 

A substantial number of evaluation criteria requirements for vocational 
education evaluation are found in this section. If pertinent criteria 
are identified in the needs assessment phase, further criteria require- 
ment identification and the development of techniques for collecting and 
analyzing data relative to those criteria will be helped greatly. 

1. What population information is available from local and county 
agencies? 

2. What information is available from census records? 

3. What information is available from the Employment Development 
Department or the Chamber of Coimierce? 

4. What procedures have been developed (or may be developed) to 
secure data from studies made by local, county, and state govern- 
mental agencies? 

5. What proportion of the dropout or educationally disinterested 
population is being served by vocational education? 

6. What proportion of the disadvantaged population is being served 
by vocational education? 

7. Are the skills of workers presently being upgraded by vocational 
education? 
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8. Are the physically and mentally handicapped being served by 
vocational education? 

9. What proportion of the total student population is benefiting 
from vocational education? 

10. How many students start work without any occupational preparation? 

11. What proportion of the student population enters college? 

12. Are sources of data on the population known to those who conduct 
the analysis? 

13. Is the population needs assessment periodic or contiguous? 

14. Is provision made for evaluating the system of .analysis in order 
to make improvements? 

15. Have individual instructors been involved in the population needs 
assessment? 

16. Has an instrument for projecting population needs been developed? 

17. Have advisory committees been involved in this analysis? 

18. How is population needs data used in vocational program development? 
' 19. What is the nature of the area served: urban, suburban, or rural? 

20. Does the analysis reflect where it may be best to teach in the 
language of monolingual groups? 

21. How is the information describing population needs kept on file 
and how is it kept up to date? 

22. Does the population needs assessment system relate on a planned 
basis with job market analysis? 

23. Is data gathered on all persons in the area to be served, including 
elementary students, high school students, postsecondary students, 
adults, migrants, and other individuals or groups? 

24. Can an exact correlation between the needs and available training be 
established? 

25. Is the cost of establishing and maintaining the system realistic in 
view of the anticipated results and benefits? 

26. Once the needs analysis is established, what will be the basis for 
establishing priorities to meet the needs? 

27. Is vocational education accessible to the entire population? 
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28. What vocational education curriculum changes have occurred as a 
result of past population needs assessments? 

29. Are the population needs files used as a.; actual evaluation and 
development tool, or are they just examples for display in required 
plans for vocational education? 

30. Is there a tendency in the population toward particular occupational 
areas? 

31. V 1 the percentage of unemployment in the district? 

3 he percentage of underemployment in the district? 

3j. possible to identify students whose academic ambitions exceed 

Lhcir ability, stamina, or fortitude to succeed? 

34. Do the results of collecting needs data indicate possible changes 
in the district's or school's philosophy and policies? 

35. Is there a constant review of labor market data and its imp i ications 
for curricu;jM development? 

36. Does the district maintain up-to-date research and studies conducted 
in the field of labor market analysis? 

37. Are the personnel involved in this function adequately trained in 
occupational analysis? 

38. Does the district validate data from other agencies by conducting 
its own labor market studies? 

39. Are labor market analysis data closely correlated with population 
needs analysis data? 

40. What information is available from private business agencies or 
organizations? 

41. How can the advisory committees participate in compiling informa- 
tion for occupational analysis? 

42. Is the school district on the mailing list of organizations and 
agencies that develop job market data? 

43. What criteria are used by decision-makers to determine the validity 
and effectiveness of the job market analysis? 

44. How are job market analysis data coordinated with job performance 
requirements analysis, to assure that both kinds of information 
are used in curriculum development and program planning? 
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45. How much of the job market information obtained from local 
industry has proven accurate or beneficial in planning vocational 
education programs? 

46. Does the secondary school cooperate with the community college in 
develo n in9 job infor -Hon? 

47. In what occupational areas are jobs developing? 

48. In what occupational areas are jobs declining? 

49. Why do current jobs exist? Is it because of expanding opportuni- 
ties, high turnover rate, or other reasons? 

50. Are the available jobs on a level that is high enough for them to 
be lookrd upon as career occupations and not merely parttime or 
stopgap jobs? 

51. Which of the available or projected jobs are available tor males? 
for fema 1 -s? 

52. Which of the jobs will enploy handicapped persons? disadvantaged 
persons? 

53. Are jobs based on government contracts or ocher tentative bases? 

54. Is the local job market steady, or is it seasonal or cyclical? 

55. Is the local job market dominated by one or a few major employers? 
or are there a large number of relatively small employers? 

56. Has information from labor unions and private employment agencies 
been included as part of the input of the job market analysis? 

57. How many vocational programs remain in operation when the job market 
analysis indicates no labor market for the occupation? (Very impor- 
tant when developing objectives.) 

53. Is the job market analysis information file used to determine the 
best begii;*ing and ending dates for programs in order to achieve 
optimum placement in seasonal occupations? 

59. In what way can job market information, job performance needs, and 
population needs be Mended :,ito useful program development data? 

60. Does the staff perform task inventories in order to determine the 
existence and the extent of the tasks for the occupations for which 
programs are preparing students (or propose to prepare them)? 

61. Are advisory committees used to analyze the task inventory and 
adjust the results into the program curriculum? 
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62. Are results of previous task analyses revalidated by conducting 
sample population studies? 

63. Are the job requirements identified according to different skill and 
employabili ty levels? 

64. What percentage of the existing curriculum was formulated years ago 
on an unknown standard? 

65. Is task analysis information available for new and emerging 
occupations? 

66. How are students informed of job performance requirements in the 
various vocational education areas? 

67. Do the skill steps outlined in the task analysis logically follow 
one another? 

68. Are subject-matter spec sts consulted? 

69. Are funds budgeted for conducting adequate task analyses? 

70. Who translates the job performance requirements into curriculum 
objectives? 

71. What additional equipment and facilities will be needed to offer a 
program based on the needs assessment, task analysis, and occupa- 
tional analysis? 

72. How are performance standards measured before completion of training? 

73. Have measurable goals and objectives been developed and published 
for previous curriculum development projects? 

74. Have evaluation criteria been developed and written for previous 
projects or programs? 

75. What method, technique, or rationale is used to determine which 
instructional units should be included in a course of study? 

76. Are the tentative course or program completion requirements com- 
patible with the needs of business and industry? 

77. In task analysis is adequate consideration given to both the 
frequency of performance and the difficulty of learning? 

78. Have complete job descriptions been developed for all occupations 
in the community for which training is provided? 

79. Are former students surveyed regarding job performance require- 
ments that they can identify? 
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80. Are entry-level job tasks separated from advanced tasks? 

81. How often does the advisory committee meet to review job 
specifications? 

82. Are present job specifications realistic in terms of learner 
abilities—especially entry-level requirements? 

83. What methods are (or may be) used to determine the qualifications 
of those who supply technical data for the survey? 

84. What control do unions or labor organizations exercise in the 
establishment of job specifications? 

85. How are preconceived ideas about required skills and knowledge 
prevented from biasing the results? 

86. Are the results of the analysis checked with other studies and 
surveys to check for discrepancies? 

87. Are the overall objectives clear to everyone involved? 

88. Are the goals and objectives realistic i terms of attainability? 

89. Are goals and objectives stated in a manner that will facilitate 
the ease and reliability of the evaluation? 



Program Planning 

The major purpose of the program planning evaluation phase is to provide 
information that will aid in the selection of a program that will best 
meet the objectives set in the needs assessment phase. The questions the 
evaluator is concerned with in this phase help determine what the program 
is or should be in order to meet those needs. 



1. Is program planning related to the needs established in the popu- 
lation needs assessment, the occupational analysis, and the task 
analysis? 

2. Does the program planning include cost estimates? 

3. Is there evidence that the recommendations of occupational advisory 
committees and occupational surveys, and other forms of advice by 
concerned community representatives is being followed? 

4. Does the planned program appear to follow a reasonably logical 
scope and sequence? 
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5. Is consideration for placement service included in the planned 
program? 

6. Is the program articulated with programs in feeder and recipient 
schools and other agencies? 

7. Do the planning activities include input from teachers, students, 
employers, and graduates? 

8. Is the planning preceded by the collection of appropriate data? 

9. Is a community or area vocational advisory committee used? 

10. Are the vocational education programs offered by neighborir a voca- 
tional education facilities considered with articulation and/or 
duplication of effort in mind? 

11. Is flexibility built into the plan in order to meet the changing 
needs of students and employers? 

12. Are supportive services adequate for the programs? 

13. What deficiencies exist that might inhibit the development of the 
program? 

14. Is area planning realistic in terms of geographic boundaries and 
job opportunities? 

15. What occupational training is being performed by recognized 
private schools in the area or other vocational programs? 

16. Is the present economic base of the community considered in pro- 
gram planning? 

17. In program planning, are provisions made for the disadvantaged 
and handicapped? 

18. Is the scope and sequence of courses for each program flexible 
enough for all students? 

19. Has a follow-up study added justification for existing programs? 

20. How is program planning evaluated? Is it built into the system? 

21. Is the proposed or existing instructional program based on needs 
of the students rather than on the training and ability of the 
teachers? 

2? Are several options considered in program planning? Have cost 
estimates been made for each alternative? 

23. Have measurable goals and performance objectives been developed 
and written for existing or proposed programs? 
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24. Do present program planning procedures involve the concepts and 
techniques of systems analysis? 

25. Do all persons concerned with the program fully understand their 
role in the system? 

26. How are priorities established relative to the use of funds for 
vocational programs? 

27. Are plans written in clear, concise language that can be under- 
stood by all parties concerned? 

28. Can the various evaluation efforts that will be most useful be 
identified? 

29. What constraints (such as unobtrusive observation) govern the use 
of the evaluation? 

30. Is the planned evaluation an integral part of the program? 

31. Are the implementation and progress evaluations coordinated with 
the outcome evaluation? 

32. Who is (or should be) involved in developing evaluation instruments? 

33. How are the philosophy and goals of the program translated into 
evaluation instruments and/or data? 

34. Does the proposed evaluation system allow for feedback or continuous 
input into the program? 

35. As a result of progress evaluations, can immediate action be taken 
to resolve conditions identified as being inadequate, ineffective, 
or deficient? 

36. Are evaluation processes established so that the information derived 
will easily facilitate the decision-making process? 

Implementation Eval uation 

The purpose of the implementation evaluation phase is to determine what 
is being done to implement the course or program as it was planned. The 
Center for the Study of Evaluation recommends that the implementation 
evaluation be conducted at least three times; before instruction, after 
instruction begins, and at the end of a major unit or component of the 
program. The evaluator questions in this section are organized around 
those three evaluations. 
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Before instruction: 

1. Has the staff been oriented to the program and materials? 

2. Are all required materials, equipment, and staff on hand? 

3. Is all equipment required for the program in operating condition? 

After instruction begins: 

1. Is the program being implemented according to the procedures that 
were planned? 

2. What are the reasons for any deviation from the plans? 

3. Do the planned procedures that are in operation appear to be 
working? Why? 

At the end of a major unit or component: 

1. Which classes o- students are succeeding? Why? 

2. What has happened in other programs at this stage? 

3. Are planned procedures being used? Are they working effectively? 
If not, why not? 

Progress Eval uation 

The purpose of the progress evaluation phase is to determine what is 
happening as a result of the program. The quests ,s that the evaluator 
will be concerned with in this phase help identify criteria that indi- 
cate how the program is progressing. The first progress evaluation is 
usually conducted at the same time as the last implementation evalu- 
ation, that is, at the end of a major unit or component of the program. 

1. How much progress toward the specified objectives has in fact been 
made? 

2. Is the progress up to the time of evaluation satisfactory? 

3. Is the program operating within budget limitations? 

4. Does the progress compare with competing or other programs in the 
area? 
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5. How can the amount of student progress be determined? 

6. Are the students ready to progress to the next unit of instruction? 

7. Could the students have moved to the next unit earlier? 

8. Have circumstances beyond the control of the program operators 
(strikes, etc.) impeded progress? 

9. Has all student progress (or lack of progress) been a direct result 
of the program? 

10. Are program planners involved in the progress evaluation? (Possible 
bias indicated if "yes.") 

11. Have students dropped out of the program? If so, why? 

12. Are the students motivated or challenged by the program? 

Documentation Evaluation 

The purpose of this phase of the decision-facilitation evaluation is to 
determine how the program was actually conducted. Documentation evalua- 
tion is. summative, that is, it occurs after the program has been com- 
pleted. The questions the evaluator asks are concerned primarily with 
what happened during the program and why. The answers to these questions 
will be helpful in interpreting the answers to the questions the evalu- 
ator asks in the outcome phase. 

1. Was the program conducted as planned? 

2. What deviations occurred, and why? 

Outcome Evaluation 

The purpose of the outcome evaluation is to measure the results or outcomes 
obtained by the program. These outcomes are by necessity related to the 
original goals and objectives although unintended results may appear. 

1. How will evaluation results be used to determine the initiation, 
maintenance, or deletion of this or other programs? 

2. What evidence indicates the degree to which the objectives have been 
met? 
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3. How are placement statistics and i!i r Jt.f • ; e.! '« t 1 "." 1 1 . u .op | 

of a vocational program? 

4 Are inferential statistical treatviv.r i.pniio' * : o-\ ;i? ia collected 
for evaluation purposes? If so, wrn<?- r-r;t-: 7 

5. Do the evaluation instruments (such r;s tests) nave empirical validity? 
(Do they really measure what is intended?) 

6. Is there a follow-up system to provide information on the effectiveness 
of the program? 

7. Did all students who started finish? Why': Were they placed early? 

8. What importance should be placed (or is placed by program decision- 
makers) on statistical data in relation to qualitative data? 

9. Does the summative evaluation involve those persons who will be 
responsible for making changes or terminating the program? 

10. What method (for example, personal observation; standardized 

written, oral, or manipulative tests; criterion measures; placement; 
follow-up) is used to measure attainment? 

1. In the space preceding each of the following decision questions, place | 
the letter of the evaluation phase in which the question is asked. 

a. Needs Assessment 

b. Program Planning 

c. Implementation Evaluation 

d. Progress Evaluation 

e. Documentation Evaluation 

f. Outcome Evaluation 



1. How much net gain did the program participants experience 
compared to previous groups? 
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2. How much progress have the program participants demonstrated? 

3. What should the objectives of the program be? 

4. Are all students ready to begin the next unit of instruction? 

5. Why are teachers not following the prepared plans in their 
instruction? 
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6. Was the program conducted as planned? 

7. Have provisions been made for continuous evaluation of the 
program? 

8. Were any progress in the past designed to meet the s^me 
objectives? 

9. Have all objectives been met? 

10. Has an occupational analysis been conducted to determine the 
efficacy cf offering the program? 

11. What practir.es occurred in the program that were not a part 
of planned activities? 

12. Was the program worth the cost? 

13. Have factors other than program activities influenced the 
results? 

14. Do all the planned procedures that are in operation appear to 
be working? 

15. Has a cost estimate or comparison been made of the proposed 
program? 



IDENTIFICATION , COLLECTION , AND ANALYSIS OF DATA 
FOR DECISION-FACILITATION EVALUATION 



Data Identi fication 

Before educational evaluators can collect data, they must first know 
and specify what they are trying to identify or measure. The process of 
specifying what is to be measured usually involves the construction and 
listing of a set of program objectives. One of the evaluators roles in 
the needs assessment phase of decision-facilitation evaluation is to 
assist the program operators do just that— establish objectives for the 
program or course. Module 7, "Derivation and Specification of Instruc- 
tional Objectives/ and several other excellent sources deal with this 
process (see Suggested References). The underlying principle of 
objectives is that they be stated in a manner that describes measurable 
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or observable behavior. This principle is illustrated by the two 
statements below: 

1. 'The student will understand educational evaluation." 

2. "The student will be able to list and state the purposes of 
the various models or conceptions for educational evaluation." 

The second statement is more likely to lead to the proper selection or 
development of measuring instruments because it clearly defines the 
kinds of evidence required in order to indicate whether or no. the 
student has acquired the appropriate knowledge. The actions required 
of f he student in listing and stating are less ambiguous than in knowing , 
and hence, are more easily observed and measured. 

Module 9, "Testing Instructional Objectives," deals with the process of 
constructing instruments for measuring the degree of achievement toward 
an objective. Several additional sources are listed in the references. 
The primary rule for test item construction is that all measures, or 
test items, be logically consistent with the objectives they are de- 
signed to measure. Each item or combination of items should elicit a 
representative sample of the behaviors specified in the objectives. 

If one objective is related to others, test items developed for that 
objective may also be used to measure the related ones. The reasons for 
measuring related objectives together include: (1) to provide infor- 
mation on unanticipated outcomes of a prog ran. . (2) to indicate how close, 
or to what degree, a program or student came to meeting or exceeding 
the objectives; (3) to provide an indication as to what level subsequent 
instruction or program activities should be aimed; and (4) to provide 
data concerning the level of difficulty or sequencing of the various 
objectives. 

Measurement instruments fall generally into three categories: (1) those 
that require selected or constructed responses; (2) those that employ 
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objective or subjective scoring; and (3) those that test and compare 
gv^ , os or individuals. Each measurement instrument developed and used 
wiM have one of the two dimensions of each of the three categories. 

The selected response/constructed response distinction refers to whether 
students select their answer from a group of optional answers (provided) 
(as in a multiple-choice question), or whether they develop their own 
answer (as in a completion or essay-type question). Selected response 
items are generally used for assessing skill and knowledge objectives; 
constructed response measures are generally used for objectives that 
focus on the generation of ideas. 

The objective/subjective scoring distinction refers to the objectivity 
of the person who scores or records the measure. An objectively scored 
measure has specific answers (provided by the test-maker) for each 
question; the subjectively scored measure relies on the judgment of the 
^ scorer. A multiple-choice item would be an example of an objectively 

scored measure, and a written essay would be an example of a subjectively 
scored measure. The objectively socred measure is usually preferred by 
decision-makers because it generally provides more reliable information. 

The third category, individual/group testing, refers to whether the pro- 
gram participants are observed or tested singly or in groups. Group 
observations are preferred for several reasons, among which are economy 
and the increased chance that all participants will be measured under 
the same conditions. 

Many methods or techniques can be used by the evaluator to gather data 
related to program objectives. (Module 9, "Testing Instructional Objec- 
tives," briefly explains the similarities, differences, and uses of 
measurement approaches, and the reference sources may be consulted if 
you desire more information.) Test items include true-false, completion, 
matching, multiple-choice, short answer, essays, and performance tests. 
^ Observation-type measurements include interviews, records and report 
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analyses, and personal observations using such techniques as question- 
naires, inventory lists, and checklists. 

Whatever measuring methods are employed, to be of use to the decision- 
maker they must meet four criteria: relevance, comprehensiveness, 
reliability, and feasibility. 

Each evaluative measure must relate to at least one of the objectives 
of the program. When a number of possible measures are available, 
the evaluator must choose the one that is most relevant to an objective 
or set of objectives. 

Evaluative measures have to be comprehensive in order to provide the 
decision -maker with useful data. "Comprehensi ve M means that the mea- 
sure must cover one or a set of objectives completely by measuring all 
the behavior indicated. In some cases, one measure can measure all 
objectives of a program, such as when a student is required to construct 
a product based on a given amount of instruction and practice. The 
product, constructed to certain specifications, might measure all the 
objectives of the program or course. 

The third criterion that evaluative measures must meet is reliability, 
that is, the measurement of what is intended co be measured must be 
precise. The evaluator has to be careful in selecting measures to 
meet the reliability criterion, because as the reliability of a measure 
increases, the relationship of the measure to the objective or objec- 
tives often decreases. The degree of reliability is usually based on 
comparisons with results of similar evaluations. 

Evaluative measures must also be feasible. Certain measures for objec- 
tives might not be feasible for reasons of cost, technical possibility, 
or ;iuman consideration. While many measures proposed for use in educa- 
tional evaluations are not feasible primarily because of excessive cost, 
in some cases human rights or comfort might be compromised, or the tech- 
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riology needed might not be available. Evaluators have to examine mea- 
sures to ensure that they are not too costly, too difficult to employ, 
or violate human rights or comforts. 

The purpose and types of measurement employed will depend on the stage 
of evaluation being conducted, but a substantial part of measurement 
data will be collected by testing students or program participants. 
(This is especially true during the formative and summative stages.) 
The interpretation and validity of the information that tests yield 
depend on the proper construction and administration of the tests. 

There are several crucial questions with which educational evaluators 
must be concerned when measuring student attainment of objectives. The 
evaluator must ask when to test or measure in order to provido decision- 
makers with the information they need most efficiently and effectively. 
Once the best testing time has been determined, the evaluator to 
ensure that appropriate locations are available for testing. T est ad- 
ministrators have to be given detailed instructions on how to administer 
the test, and student, or program participants must 4p informed of the 
purpose of the test and given the directions for completing it. 

Data Collection (3) 

It is useful to examine the three main components of- the process of data 
collection: data sources, means of tapping the sources, and data col- 
lecting agents. 

Data Sources . The sources of data are either people or recording devices. 
To learn about the behavior or characteristics of indi viduals, who have 
participated in a program, or in the processes of the program, one could 
consul t: 

1. the individuals involved; 

2. teachers, supervisors, or others involved in designing or 
carrying out the program; or 
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3. parents, co-workers, or others in a position to observe or 
be affected by the behavior of the individuals or the pro- 
cesses of the program. 

Alternatively, one could design a device to record or count instances 
of relevant behavior. In most cases, it is best to assume bias on 
the part of any person used as a data source and even op the part of 
the automatic device, since it had to be designed by an imperfect human. 
The evaluator should usually choose a type of data source, however, 
that is assumec to have the least bias, or one that is in the best 
position to observe, or one that has the specialized knowledge necessary 
for the required data collection. And in the majority of cases, it 
is best (if feasible) to employ several data sources, so that their 
various biases and inaccuracies can be compared and at least partly 
understood. 

Means of tapping sources . A given data source can normally be approached 
in a variety of ways. People may be asked questions in an interview or 
in a questionnaire (or in that special form of questionnaire, the 
test). People or their traces (the presumed consequences of their 
actions) may be observed and records of various kinds maintained by 
the observer. Observations may be made in natural surroundings or in 
simulated situations. The observers may be passive and simply record 
what happens, or they (or associates) may interact with the object of 
observation. The individuals may know that data are being collected 
about them and to some degree may thereby be influenced in the data 
they provide, or they may be unaware and the measurement be unobtru- 
sive or nonreactive (31), with the data presumed uninfluenced by the 
individual's awareness. In some cases, the data will have already 
been collected and stored, so that the collection effort becomes a 
matter of retrieving the particula, items of interest from the file, 
computer tape, or other repository. Since each variation in eliciting 
data has its limitations, the gv >ater variety of means used to tap 
data sources, the better. 
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Data Collectors . Different people, and sometimes different types of 
automatic recording d2vices, can collect different data from the same 
source. Although it is important to select and train data collectors 
to be objective and capaole at the tasks they are to perform, there may 
sMll be important variations in the data they collect. Some reasons 
for the variations can tentatively be identified. It is easy, for ex- 
ample, to think of studies in which the racial appearance, sex, length 
of hair, or appart-::- age of an interviewer almost surely influences the 
results. In other studies the reasons may be so subtle in origin as to 
be difficult to detect and therefore all the more important to guard 
against. In any event, it is desirable, if feasible, to employ several 
data collectors. 

Indexes of Outcomes 

For purposes of discussion, let us distinguish between outcomes in 
v h we are interested and the indexes we use to represent or measure 
those outcomes. Three classes of indexes will be discussed: behavioral 
measures, other outcome measures, and process measures. 

Behavio ral measures . If a program is designed to modify behavior, then 
it seems logical for the evaluation measures for that program to measure 
)K>h vior. To a great extent that logic is correct. When it is possible 
u state program objectives in behavioral terms, it is best to determine 
as directly as possiblo ther the behaviors in question have appeared, 
disappeared, or changed at* specified. Note that in such a case it is 
unnecessary to collect information about other behaviors. It is also 
unnecessary in any but exceptional cases to identify degrees or levels 
of behavior. The need rather is to determine whether a specific behavior 
did or did not occur--a simple, two-p-rt or dichotomous classification: 
a student did or did not volunteer to participate 1.1 a class activity; 
a smoker did or did not refrain from smoking in a given time period; a 
tuacher did or did not praise an appropriate student behavior. In many 
cases it is also important to learn the number of instances when a behavior 
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appeared or did not appear; thus, the final index might be the propor- 
tion of times, of all the times when the stimulus conditions were ap- 
propriate that a given behavior occurred. 

Other outcome measures . Outcome measures of nonbehavioral types may 
be appropriate under two main conditions. The first condition arises 
when it has not been possible to state outcomes in specific behavioral 
terms. Suppose, for example, that the desired outcome of a program is 
the improvement of attitudes toward persons of diverse ethnic and racial 
backgrounds. Such attitudes are presumably important to the extent that 
they are reflected in behavior, and it could therefore be argued that 
it is best to measure the real behaviors rather than the hypothetical 
attitudes. However, this approach is not often feasible, and it is 
necessary, therefore, to ask people directly or indirectly to describe 
their attitudes. The second condition under which nonbehavioral outcome 
measures are appropriate is when the behavior leaves some product or 
other trace that can be measured. An anti -1 ittering campaign could 
be evaluated by counting or weighing the amount of litter in a given 
area before and after the campaign. 

Note that both these conditions may also prevail when behavioral measures 
are appropriate and feasible. In other words, nonbehavi^ al outcome 
measures can be supplementary a< well as alternate to behavioral mea- 
sures . 

Process measures. A process measure is one that describes something 
about the program, rather than about its results. Examples of process 
measures are attendance counts, numbers of hours of instruction, degree 
of adherence to planned schedule and agenda, judgments of effectiveness 
of speakers and audiovisual aids, adequacy of facilities and control 
of environment, and judged receptivity and interest of participants. 
Process measures are generally poor substitutes for outcome measures. 
However, there are occasions when a process measure is appropriately 
used, either because of its unusual merit in a particular situation 
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or because obtaining the appropriate outcome measure is not feasible. 
Suppose, for example, that measurement of the effect nf , film presen- 
tation is difficult in a particular situation, but c-»re -.re experts 
available who know the effectiveness of various type*: of films in similar 
situations. In such a case it might be preferable to use the experts' 
judgments (a process measure) rather than an unreliable or biased out- 
come measure. On the other hand, the best strategy might be to use 
both an outcome measure and a process measure. 

It is conceivable for a process measure to be the equivalent of an out- 
come measure in certain situations. Consider, for example, a program 
that has as one objective the stimulation of continued interest on the 
part of participants in the program topic. Continuing attendance at the 
program sessions, measured in a process way, might well be a good be- 
havioral outcome measure of the interest of the participants. 

It should be mentioned also that process measures serve another purpose. 
They describe the program (the independent variable) as it was actually 
presented, rather than as it may have been planned or envisioned, and, 
therefore, they are an initial basis for explaining whatever effects 
may be found through the evaluative measurements (the dependent variable). 
For this reason it is generally desirable to collect a rather comprehen- 
sive set. of process measures, especially during the < progress and documen- 
tation phases of decision-facilitation evaluation. 

Data Analysis (27) 

After the evaluator has been properly -oncerned about instructional 
objectives, devised suitable ways for measuring them, and skillfully 
designed schemes for gathering the data yielded by these .measures, the 
data must still be analyzed. These are a few points wHch can help him 
in this task. 
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A nalysis Unit s 

In most types of data analysis we compute descriptive statistics, such 
as the mean, median, and standard deviation, in order to economically 
describe the data under consideration. The units on which these numeri- 
cal indicators should be calculated are the smallest independent units 
available. This is definitely not always the individual learner. For 
example, if five classes with 30 students in each are given treatments 
which are heavily dependent on the teacher as well as the classroom 
interaction of the students, then we should not calculate a mean based on 
the performance of 150 students but, instead, a mean based on the average 
performance of each of the five classes. In other words, our datum 
( data : plural) here would be the mean performance of a class sin'e that 
represents the smallest independent sample available. 

In this example, the evaluator should regard his measurements as 
providing five pieces of data regarding the success of his program. He 
might compute the mean (or median) "score" for each class and present 
these as five pieces of data--and he might even compute the mean or median 
of these five. Even though the mean of all the students' individual 
scores might not differ much from this latter value, to compute that 
figure is conceptually in error. And should the evaluator go ahead to do 
significance testing or inferential estimation, he needs to base his 
calculations on five independent units and not !50. Things aren't as 
bad as thev seem, since means are much more stable than individual 
scores and you may gain power, not lose it, by analyzing five stable means 
rather than 150 highly variable scores. 

On the other hand, if wo were using the same five classes but were now 
evaluating individual sol f-instruction booklets which took total respon- 
sibility for promoting learning (with no teacher or classmate interaction), 
then it v/ould be legitimate to compute descriptive statistics on the 
entire group of 150 learners since the datum from each pupil wouid be 
the smallest; independent unit:. 
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In spite of the fact that educators have for years conducted their data 
analyses on the basis of the total group of learners, the analysis unit 
approach described here has been strongly recommended during the past 
several years by leading research design specialists and should definitely 
be used by the educational evaluator. This means that in constituting 
groups by randomization, as in some of the control group designs we 
examined earlier, the units randomly assigned to one group or another 
might be classrooms (or even schools or school districts) rather than 
individual learners. 

Thus, if an evaluator finds himself in a situation where practicalities 
of the ongoing educational program preclude the random assignment of 
individual pupils to various treatment conditions, the assignment of 
classroom units by random may be feasible. 

Estimation Versus Hypothesis Testing 

Most educators who have completed the customary statistics courses 
have encountered a variety of techniques with which to analyze da.a. 
Such statistical tools as the t test, correlation coefficient, and 
analysis of variance have typically been mastered by inc. ■•-i.-Ulonal 
evaluators. Unfortunately, because these analytic tools tfiuiin 
their repertoire, some evaluators always try to apply '.tan ti- :he 
analysis of performance data in evaluation studies. r n smik /ays, 
these techniques are unsuitable for this purpose. 

The majority of statistical technioues commonly seen '• ir research 
journals are hyjjothesis testing procedures and are dc _.ned to reject 
null hypotheses concerning the existence o r a relation^? between two 
variables. In a commonly seen situation, ft\ example, the posttest 
mear performances of > .io groups are contrasted to see if the mean of 
t> ; Method I group is significantly higher than the mean of the Methcd Y 
iroup. When a t test is applied to such data the question it attends U 
answer 1'. the following: "I there a reliable difference between tho 
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two groups?" If there is a reliable difference, then a relationship 
has been discovered between the method variable (X versus Y; *nd 
the posttest variable. 

But evaluators are rarely interested only in whether t^r* ,s a 
difference between two or more treatment groups. They ^'*e also 
concerned with the magnitude of the difference. To yind an indica- 
tion of how much difference exists between groups, hypothesis 
testing techniques are not suitable. 

Instead, a group of statistical devices known as eoti n a tion procedures 
can be used to supply us with an approximation, a rather precise 
one in some cases, regarding the magnitude of treatment differences. 
To illustrate, if we have detected on the basis J a particula. 
evaluation study that Treatment A is 4.2 mean points better 
Treatment B, by establishing a confid ence interval we c;; identify 
a range of mean differences which at a giver probability ".fal. for 
example, with odds of 95 to 5, would include t u e true (poutil^cion) 
mean difference- For example a 95 per cent confidence \\- Nerval of 
mean differences between 3.1 and 5.3. More strlr.yent confidence 
internals, Per example, 99 per cent can also cal "ulated. 

If the evaluator is not familiar with the pmr retires for computing 
confidence intervals and similar estimation indices, he may wish to 
consult a recent statistics textbook designed for use in the behavioral 
sciences. l'\ on the other hand, a statistical consultant is called 
in to assist in the data analysis, the ? alitor should discourage 
hi g exclusive use of hypothesis testing procedures and explore,, 
intend, the suitability of estimation procedures. 

In reporting results of learner performance on the various measures 
of interest, the avaluator will discover that simple descriptive 
statistics wiM communicate with a wider audience than many esoteric 
statistical analyses which might be used. For instance, it may be 
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sufficient to merely supply (1) the average percentage correct or 
(2) the proportion of learners reaching the desired criterion. The 
evaluator is often less interested in subtle differences that have to 
be teased out by sophisticated statistical techniques than he is in 
major magnitude differences that c<n be readily communicated to educational 
decision-makers. If hypothesis t--'t^Vj techniques must be used, then 
one of the many nonparametric te-'.'a may be suitable particularly in 
view of the case with which they &ra calculated. The most readable 
description of nonparametric statistical procedures is still that 
prepared some years ago by Siegel.* The evaluator will find the Siegel 
text useful . 

Decision-Making in a Cost/Effectiveness Context 
The educational evaluator typically is not the final decision-maker 
regarding the alternative courses of action associated with either needs 
assessment or treatment adequacy assessment. Generally, he will be 
supplying information to others who will make these decisions. He should, 
however, present a wide range of information so that these decision- 
makers can make wise choices. 

Although the use of instructional objectives has been stressed throughout 
the guidebook, it must be emphasized that mere attainment of objectives 
is insufficient for the evaluator to reach a positive decision. For 
instance, suppose an instructional treatment proved effective in promoting 
a set of objectives but only with a financial expenditure far in excess 
of what could be afforded. Obviously, the decision-maker would not opt 
for the prohibitively expensive treatment. Putting it more generally, the 
evaluator should supply sufficient information so the decision-maker 
can contrast costs with effectiveness prior to reaching a conclusion. And 
the costs involved are not only financial. We can conceive of some 
otherwise effective instructional treatments which might be inordinately 
costly in terms of teachers' lowered morale. 
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*Siegel, Sidney. Nonparametric Statistics for the Behavioral Sc iences, 
McGraw-Hill, New York, 1956. 
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Or we might think of needs assessment evaluations which would identify 
certain objectives as having the highest priority on the basis of 
preference data and discrepancies between current and desired learner 
performance. Yet, to implement treatments designed to accomplish 
these objectives may simply be too expensive. A cost/effectiveness 
decision will be different than a decision based on non-cost factors. 

The significance of this point is that the evaluator should present 
decision-makers with as much information as they might reasonably use 
in deciding among alternatives. We have seen previously that data 
regarding learner performance on objectives is crucial. Preference 
data which can be translated into value standards will also prove useful, 
as is evidence of unanticipated outcomes associated with any instruc- 
tional treatment. Now we have suggested that cost data are also 
requisite. The more relevant data, the more effectively the educational 
decision-maker can function. 

In reviewing this section on data collection and analysis we have 
examined schemes for (1) gathering preference data and (2) comparing 
preference and performance data. We have also seen (3) how item and 
person sampling can conserve data gathering time and which types of 
designs are particularly suitable for (4) formative evaluation and 
(5) summative evaluation. In addition, we have examined (6) the 
analysis units to be used in treating evaluation data, (7) estimation 
versus hypothesis testing procedures, and (8) decision-making in a 
cost/effectiveness context. 

Popham's three guidelines for data analysis *ire: 

Guideline Number 18. The educational evaluator should analyze data 
according to the smallest independent units available, frequently 
leading to the use of classroom or larger units rather than individual 
pupil units. 



ERLC 



77 

-66- 



Guideline Number 19. The educational evaluator should, in general, pre x 
descriptive statistics and estimation procedures instead of statistical 
hypothesis testing procedures. 

Guideline Number 20. The educational evaluator should present decision- 
makers with a wide range of pertinent information so that choices 
among alternatives can be made in a cost/effectiveness context. 
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In the space preceding each of the objectives below, indicate what type 
of data would be collected by the evaluator in order to measure the 
extent to which the objective had been or is being achieved. The 
objectives below have been excerpted from several local district plans 
for vocational education and from vocational program or course objectives. 



The data types are: a. behavioral change indices 

b. attitudinal change indices 

c. process change indices 



1. To expose students to an information and recruitment 
program for vocational education in grades 7-12. 

2. The students will develop an appreciation of mechanical 
skills. 

3. To provide a continuous evaluation of the students, the 
instructional program, and of the community. 

4. To develop career consciousness as an integrated part of 
the students' experiences in order to enlarge the number 
of options and alternatives for them, both in terms of 
occupations and further education. 

5. The student will be able to correctly survey a traverse 
using a transit and tape. 

6. To identify those individuals not now being served by 
vocational education and to encourage them to enroll in * 
program of instruction. 

7. The student's awareness and appreciation of the role of 
gainful employment in society will increase. 

8. The program will function properly at the institution or 
with the group implementing it. 

9. The student will be able to stay-stich the curved neck 
edge of a bodice. 

10. The student, after completing the career awareness program, 
will desire to obtain further education relative to his 
personal goals. 
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Select from the list provided the type of instrument or technique 
the evaluator might best use to obtain answers to the following 
questions and indicate it in the space preceding each numbered question. 
The questions may be instructional objectives or program objectives. 
Instruments or techniques may be used more than once. 



Data collection instruments: a. 

b. 
c. 
d. 
e. 

f. 



anecdotal records analysis 
personal observations or checklists 
interviews 
questionnaires 

tests (i.e., multipl choice, true- 
false, etc.) 

performance examine ju: (doing 
something) 



f . 1. Will the student be able to properly remove and replace 
nuts and bolts with an air wrench? 

2. As a result of this program, will teachers of elementary 
~~ grades have a more positive attitude toward the career 
education concept? 

3. Does the school board support the idea of expanding 

vocational education offerings in the school? 

4. Will potential dropouts remain in school as a result of 

this program? 

5. Do the program's objectives conform to legal requirements? 

6. Will the student be able to correctly obtain an accurate 

reading of a patient's blood pressure? 

7. Will the student understand the physiological reasons for 

high or low blood pressure? 

8. Which students benefited most from the program? 

9. What proportion of the population in the community is being 
~ served by vocational education? 
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10. What percentage of the students in a given school might be 
considered disadvantaged? 

11. Are program operators conducting the program as it was planned? 

12. Will the students be able to identify the qualities of a good 
employee? 

13. Will the students be good employees upon completion of the 
program? 

14. Is the program treating each of the specified objectives? 

15. Are goals and objectives stated in a manner that will facili- 
tate the ease and reliability of the evaluation? 

16. Are all personnel involved in the program satisfied with the 
way it's being conducted? 

17. Are the skills the students will obtain at the end of the 
program compatible with the needs of industry or business? 

18. Will the students possess all of the skills required of them at 
the end of the program? 

19. Are supportive services adequate for the program? 

20. How frequently do job holders perform certain tasks in any 
given day? 
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Place a check mark (7) in the space provided only if the question 
refers to an evaluation data requirement for the particular phase 
under which it is found. 

Needs Assessment 

1. Does the district conduct a planned program of student 

recrui tment? 

J_ 2. Have measurable performance objectives already been written? 

____ 3. Is there a schedule for routine review of the program? 

4. What is the percentage of dropouts in the high schools of 
the district? 

5. Are texts and reference materials current in their content? 

Program Planning 

6. Do instructors regularly participate in in-service training 

programs or workshops? 

7. What articulation agreements are in effect? 

8. Is existing equipment modern and in good repair? 

9. How many students have requested vocational programs? 

10. Have students shown any growth as a result of the program? 

Implementation Evaluation 

11. what is the basis of the education being provided: student 

needs or employer needs? 

12. How many of che students are showing cognitive growth as a 

result of the instruction? 

13. What r::-.atistical techniques should be u^ed to analyze 
— student performance data? 

14. Are planned procedures working effectively? 

15. Is the staff ready to conduct the program? 



Progress Evaluation 

U. Have students achieved the specified objectives? 

_ 17. Are students interested in continuing in the program? 

18. Do the teachers need he 1 p in yetting any of the components of 

the program in operation? 

19. Have all the materials arrived? 

20. Is it important that this report be sent to all persons 

responsible for the program? 

Documentation Evaluation 

21. What evidence is there to the degree to which objectives were 

met? 

22. Was the program conducted exactly as planned? 

23. Are the students ready to leave the program 9 

24. What deviations from the plan were there in the conduct of the 

program? 

25. How much did it cost to operate the program? 



Outcome Evaluation 

26. How much progress did the students make? 

27. Which statistical methodology should be used to analyze student 

' ~ performance data? 

28. Was the program conducted as planned? 

29. Should the program be continued? 

30-. Do program results justify costs? 
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Answer the following questions: 

a. What are the two types of procedures commonly usee; in statistical 
analysis of data? 

b What is the main difference between statistical analysis for 
educational research and statistical analysis for evaluation 
purposes? 
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Goal 15.4 



Content Outline 



^VGoal 15.4: Know the Methods for Pre- 




Activities-Resources 



Preparing an Eval ua tion Plan 

The evaluation p^n is 'sually submitted with an 
original proposal for a project or program and is 
an integral part of that proposal. The plan 
indicates how and at what times the program will 
be evaluated and what decision needs will be 
served. For decision-facilitation evaluation the 
plan will specify how, when, and why certain 
activities will be conducted during each evaluatioji 
phase. A flow chart indicating the activities and 
their timt relationships is often included in the 
evaluation plan. 



R eporti ng Times 

In the plan, the evaluator specifies when reports 
will be submitted to the decision-makers. He basei 
the timing of thes^ reports on when they will most 
effectively help in decision-making and determines 
this by observation and communication with the 
decision-makers. Reporting is always done at leas'; 
once--at the end of a major stage—but reports wi 1 
also usually be required at several other points 
during the course of a program, especially if it is 
a long one. 



Content Outline (continued) 



< 



C. evaluation Report Conteirt and F_qrinat 

1. Hawkr-.Jge, Campeau, and Trickett recommend that 
evaluation reports contain at least five 
sections. These sections are: 

a. a summary, 

b. a description of the context in which the 
evaluation took place, 

c. an explanation of the program, 

d. a report of evaluation results, and 

e. recommendations ( 15) . 

2. Although the reports made f^r each o* the ^ix 
phases could include all five of these sections 
it is more common for only the fi nal report to 
contain all five. The reports made for the 
various phases of evaluation, usually called 
interim evaluation reports , g3nerally contain 
only th^ information necessary for making 
decisions. Most of these contain a brief 
explanation of the context in whicii the avalu- 
ation took place, a report of the evaluation, 
end evaluate recommendations. 

3. The five major sections of final evaluation 
reports are described below. Those items often 
included also in interim evaluation reports are 
indicated with a double asterisk (**) and the 
evaluation phase in which they are included is 
indicated in the Activities/Resources column, 
a. Summary** 

This section provides a brief overview of 
the program or phase being evaluated and 
includes: 
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Preparing Evalu - 
ation Re ports : 
A Guide for 
Authors . 



Usually precedes 
each evaluation 
report, interim 
and final .) 
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Content Outline (continued) 



( 1 ) major objectives ; 

(2) context in which the evaluation was 
conducted; 

(3) outline oV evaluation techniques 
employed; 

(4) evaluation results; and 

(5) conclusions and recommendations. 
Context ** 

This section describes the envv> ,nv mt in 
which the program took place and > slides 

(1) geographic locale and any of it' 
characteristics important to the 
evaluation; 

(2) system or institution and any o" tv ( 
special characteristics that would bo | 
of use to future planners; 

(3) needs or justification for the pro- 
gram or the particular part being 
evaluated; and 

(4) constraints such budgets, political 
atmosphere, etc.. 

Explanation of the program * •* 
This section, v^ch is usually one of the 
largest in fi . . i evaluation reports con- 
tains such items as: 

(1) scope of thfi program* including objec 
tives, the people involved, and cheir 
characteristics; 

(2) description of program personnel, 
including their roles, their expertise* 



(** Included ;.. ill 
reports, ' : Mm 
and f in* : . 
implemevito. 
and doc .^e ,L t -c : 
phases ; >' 
primarily descrip- 
tions oT che 
context ) 



r ** 



Among interim 
reports, this sec- 
tion is primarily 
included in the 
incrementation and 
documentation phases, 
but there might be 
instances when it 
would be included 
also it. progrc : 
p"! arjpi ng and p rrgress 
evaluation phases. ) 
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Content Outline (continued) 



duties, responsibilities, rec- siting, 
training, and retention problems; and 

(3) evaluation procedures, including the 
time period covered by the . eport, 
descriptions of program and p^aluotion 
activities, equipment and materials 
required, cost-benefit data, and 
budget information. 

Evaluation report** 

This section of the total evaluation report 
contains information concerning the achieve- 
ment of program objectives. Information 
on the processes and products of the 
program are included in an attempt tc 
clarify exactly what happened and why. 
Major subheadings of this section include: 

(1) program objectives (specific); 

(2) procedures and techniques for selec- 
ting program participants; 

(3) data collection techniques; 

(4) data analysis procedures and 
techniques; 

(5) report of the results of data analyst 
and 

(6) conclusions reached as a result of 
data analysis. 

Recommendations** 

All recommendations made by the evaluator 
must be supported by collected and analyzed 
data. Recommendations to program decision- 
makers are just that—suggestions, not 
mandates They should be phrased like a 
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Content Outline (continued) 



suggestion and the rationale behind them 
should be included. 
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D. Stud y Activities 

:\itj*\l en yum* veadi >:>j •*>/ the c , o>ite>it outline and tJu : - in fan; % ion yvo- 
\ y :dei \:i:r: the $ + uJy ,;o: : in t:cr> 9 ,-emrletc the. activity that follows. 

PLANNING AND REPORT ING EDUCATIONA L EV ALUATIONS 

Although planning and reporting are activities .-hat occur at opposite ends 
of the evaluation continuum—planning being tise first thing the evaluator 
does and reporting the last—reports are nevertheless dependent to a high 
degree on what was planned. What does the evaluator report, and when is it 
reported? The basic answer, of course, is that the evaluator reports 
information to the decision-maker that will meet the needs of that decision- 
maker at the time the particular decision is to be made. But more parti- 
cularly: What are those decisions? When are the reports needed? In 
what form does the decision-maker need the information if it is to help 
him make the proper decision? 

What are Decision Requirements? 

This question is the first one the evaluator asks before planning 
the evaluation. The overall purpose of decision-facilitation evalua- 
tion is to improve the educational process. This improvement is aided 
by the evaluator' s assistance in selecting objectives and determining 
the adequacy of the treatment provided by the program. The first 
tasks of the evaluator, then, are to determine from the decision-maker 
the criteria acceptable for program objectives and then provide that 
decision-maker with information relative to tho e criteria, (This step 
is part of the needs assessment phase of the prcformative sicq^ of 
decision-facilitation educational evaluation,) Similar decision 
requirements are identified in consultation with "he decision-maker for 
each evaluation phase pnor to implementation of the program. Decision 
requirements may provide the framework for the evaluation plan and the 
subsequent reporting of evaluation data to the decision-maker. The 
previous material in this guide details many of the decision require- 
ments for each phase of decision-facilitation evaluation, 
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When are Evaluation RjJ>orts Needed? 

In most instances, evaluation reports are required only at the end of 
each phase of the evaluation process, but in some cases reports may 
be required at other times. The exact schedule for delivering 
evaluation reports must be determined before implementation of the program 
and they always must be based on meeting the decision-maker's needs. 
If decision requirements serve as the framework for evaluation plans 
and reports, the scheduling of these reports is a second aspect to be 
included in the plan. 

What Form Should Evaluation Reports Take? (10) (26) 

Evaluation reports follow the form specified in the evaluation plan. 
Hawkridge, Campeau, and Trickett (15) specify five sections of evaluation 
reports: (1) the summary, (2) the context in which the evaluation took 
place, (3) an explanation of the program, (4) a report of the evaluation, 
and (5) evaluator recommendations. 

The Summary 

The purpose of the summary of the evaluation report is to provide a quick 
overview of the program. The summary includes the major objectives of 
the program, the context in which the program was conducted, and a brief 
description of the methods employed in the program. 

An outline of evaluation techniques that were employed, the results of 
the evaluation, and conclusions and recommendations derived from the 
evaluation are also included. The summary section of the evaluation report 
should be no longer than a page or two. 

Describing the Context 

Context refers to the environment in which the program took place. The 
environment might include such things as the geographic area, attitudes 
of community and school personnel, restrictions and constraints, trends in 
^ the district or school before the program, and other special characteristics 
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of the program or the district in which it was operated. The description 
of the context provides the decision-maker and other readers with a basis 
for the rest of the report. Context description is particularly useful 
to planners of future programs, as it allows them to assess the results of 
the present program in light of all factors that influenced it. Context 
description may be subdivided into three areas: (1) the locale, (2) the 
system or institution, and (3) other special factors. 

(1) The Locale . The locale description contains such items as physical 
location, population density, unemployment rates, etc. The geographic 
location description should include characteristics only of the area 
the program was designed to influence, such as rural, urban, suburban 
or inner-city. Population patterns such as density and mobility are 
included to further clarify the location factors. Economic patterns 

of the locale are especially important to vocational education programs. 
Unemployment trends should be described, as should the major occupational 
categori es. 

(2) The System or Institution . Decision-makers are obviously aware of 
che factors described in this section, so its importance lies mainly in 
serving future program planners or funding agencies. If the program 
that was evaluated involved one school, a system of schools, or several 
systems, the description should include information on the grade levels, 
number of students, rate of transiency or dropout, cost of education per 
student, and other pertinent data about the financial status or history 
of the system or institution. 

(3) Special Factors . This category should contain the justification 
for conducting the program in the first place. If the evaluator was 
involved in the program from -its inception, this information would be 
first hand, but if the evaluator was hired after the program was 

begun (as is the case all too often), this information would be available 
in the original proposal. In any case, the description should include 
a delineation of the needs for the program and how ard by whom they were 
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identified. The description of the context should also include a brief 
history of the program, including answers to questions such as how was 
it conducted, what preceded it, how was it originated, and what special 
problems were encountered in gaining acceptance of the original plan. 

Explanation of the Pr gram. This section cf the evaluation report 
is often the largest. Included in it are discussions of: (1) the scope 
of che program, (2) the people involved in conducting it, (3) procedures, 
and (4) costs. 

(1) Program Scope . What were the specific objectives of the program? 
How many students were involved? What were their ages anu grade levels? 
Were they average, underaverage, or overaverage achievers? Poor or wealthy 

(2) Personnel . How was the program staffed? What were the qualifications 
and expertise of program personnel? Were the personnel all instructional? 
Who planned? Who were the administrators and support personnel? It is 
helpful here to include short descriptions of the duties, activities, and 
responsibilities of the various program personnel. Any problems in 
recruiting or retaining personnel siiould be described. Again, th's 
section is of particular help to future program planners, and alsv may be 
of use in interpreting final evaluation results. 

(3) Procedures . In this section the evaluator indicates the time period 
covered by the present report. For which phase of the evaluation model is 
this report? What are the inclusive dates? What portion of the entire; pro 
gram evaluation is reported? Where were the activities reported herein 
conducted? Were any special physical arrangements necessary? Had the 
program been modified as the result of any previous evaluation reports 

or recommendations? Was progress toward specific objectives reviewed 
by the program staff in any way? Were deficiencies identified and 
remedied? 

Trie procedures section also contains descriptions of all activities 
conducted in the program and descriptions of tho objectives for which 
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the activities were designed. What methods were used to conduct each 
activity? How much time was devoted to each activity? How were 
activities sequenced? What feedback mechanisms (for students, teachers, 
and decision-makers) were employed? How were students motivated and 
rewarded for their efforts? 

The program explanation section also contains lists or descriptions of 
equipment and materials that were used. This would include any special 
materials and who developed them using what methods or techniques. How 
were the materials used in conjunction with the activities described 
previously? Did students or program participants from comparison groups 
have or use the same materials? 

What role did parents, advisory groups, or business-industry representatives 
have in the planning and operation of the program? How were the above 
groups kept informed of program events? 

(4) Costs . The final section of the program description contains informa- 
tion related to the costs or the budget of the program. Information that 
should be included includes funding sources, total costs, time period 
during which the funds were used, and per pupil costs. Program costs 
should be broken down by type: start-up costs, maintenance costs, 
continuation costs. 

Popham (26) has added another dimension to costs in his analysis. He 
feels that it is also helpful to analyze and report costs in terms of 
benefits forsaken and benefits gained. The benefits forsaken category 
includes the options that were given up as a result of the decision to spend 
the money as it was spent; this category describes the cost of the program 
as "the highest valued opportunity necessarily forsaken" (26, d. 259). 
The benefits gained category includes monies returned or saved as a result 
of the program. An excellent example of the latter concept is provided by 
Ghazalah (11). 
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Total costs should be described by categories, and comparisons should be 
made with other programs and normal operating costs in the school or 
system. The readers of the report should a'so be informed of the loca- 
tion of more comprehensive budget data than was provided. 

Reporting the Evaluatio n. This section of the evaluation report 
includes evidence that the program has or has not reached its objectives, 
dialogue or evidence relating to the fact that objectives were or were 
not achievea, and a basi'j for conclusions and recommendations for 
improving, maintaining, or terminating the program. 

Depending on which phase of the decision-facilitation evaluation 
model is being reported* the evaluator will prepare different types of 
information. Two types of information are typical: process informa- 
tion and product information. 

Process evaluation reports describe the extent to which the program 
was implemented as planned. These evaluation reports occur at the 
implementation and documentation phases of the CSE decision-facilitation 
evaluation model. Sources for the description of the activities include 
personal observations and records of equipment and services provided 
to the program. Generally, the purpose of process evaluation reports 
is to provide a description that will serve as a basis for interpreting 
the product-type evaluation information. Process evaluation reports 
generally do not provide a basis for determining quality of the program, 
nor do the) jsually provide any basis for conclusions or recommendations „ 

Product evaluation reports contain evidence relative to the quality of 
the program. This type of evaluation repor* occurs at the needs 
assessment phase, the program planning phase, the progress phase, and 
the outcome phase. (Actually, the needs assessment and program planning 
phases often contain both process and product evaluation information.) 
Product evaluation reports provide information that will aid the decision- 
maker in deciding to altar, maintain, or terminate a program. There 
are three general reporting areas for both the process and product 
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evaluation types. The first (a) area contains the specific objectives 
that are being reported, the second (b) area contains information about 
program participants and the third (c) area contains information on 
data collection, analysis, and reporting. 

(a.) Objectives. This subsection of the evaluation report contains a 
list of the specific objectives of the program. It is helpful to divide 
the objectives into the three classifications of cognitive, affective, 
and psychomotor. This exercise helps the evaluators clarify in their 
own minds the answer to the question: What do students now know, think, 
or do that they ch'dn't before the program? 

(b.) Participants . This subsection contains a description of procedures 
and techniques that were employed in selecting program participants. If 
a control group was used, its characteristics are described. The extent 
to which program participants were involved in other similar programs 
should be documented. The number of ogram dropouts and their reasons 
for dropping out should be listed, and any replacements for dropouts 
should be described. All characteristics of program participants should 
be described, including attendance, their commitment (voluntary or com- 
pulsory), how many and which participants received the program, and 
ages, sex, and any other special characteristics. 

(c.) Data Collection , Analys is, and Reporting , • These three areas are 
often included in the evaluation report as separate chapters or sections. 
The data collection section reports the measures that were applied to 
find out whether program objectives were achieved. Specific measures 
for each objective or group of objectives are described and often in- 
cluded in the report as appendices. The use of specific measures 
for objectives or students should be justified. The qualifications of 
the observers or evaluators should be listed and the times or intervals 
of testing or observation should be included. Once the procedures and 
techniques for collecting the data have been described, the actual data 
should be presented in a format that is easily understood by both pro- 
fessional and lay people. 
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i he next section describes the analysis methods used to examine the data. 
Aqain, specific analysis techniques should be justified for each type or 
group of data, and the basis for judging progress or quality should be 
specified. Comparisons should be made with other groups or subgroups 
of data in the present program, and evidence (if there is any) that 
program participants gained from the program should be presented. 

An important aspect of the report of evaluation findings is that all 
information be presented in simple ordinary language. Abbreviations 
often lead to misunderstanding on the part of people not familiar with 
the program. All narrative and graphic descriptions should point to, 
or clearly indicate, the success or failure of the program. If the 
findings of the evaluation indicate that the program is generalizable 
or replicable, it should be so stated. It it is doubtful that the 
program has uses in types of groups different from those who v^-ally 
participated, that should also be stated. Conclusions must bt succinct- 
ly stated and supported by existing (and reported) data. 

Recommendations . The final section of the evaluation report would be 
evaluator recommendations. These should be supported by data and by 
the conclusions that were reached and stated earlier. Any recommenda- 
tion should be referred to the conclusion that serves as its basis. 
Recommendations should be spelled out clearly and attractively displayed. 

1. For each of the following statements, indicate the evaluation 

report (by phase) in whici .he statement would most likely appear 
for a decision-facilitation evaluation. In some cases, the state- 
ment might be employed in two of tK*i p.iases, such as progress and 
outcome . 

a. Three available pre-packaged prog »m; were examined in an 
effort to find one that included • project objectives, 
program planni ng 
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Only D.io-third of the materials required to conduct the first activ- 
ity were on hand when instruction began. 

The recommendations of the advisory committee were all translated 
into measurable objectives for the program. 

No statistically significant differences we^e found between program 
participants and control group members . 

It was decided that parents and program sponsors would be advised 
of progress in monthly newsletters. 

All students had achieved the first four objectives by the end of 
the first month of instruction. 

The forty-seven objectives were then rated by the members the 
advisory committee in an effort to determine the most imporv c 
twenty . 

It is apparent from an analysis of the data that tne program must 
be modified to include . . . 

The academic subject area teachers readily accepted the added re- 
sponsibility of including the career-related concepts in their 
instruction. 
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The following procedures that were originally incorporated in 
the program plan were completely ignored . . . 

The average score on the test for the first unit of instruction 
indicates that the students have improved on all objectives except 
number three. 

It was decided that e nonequi val ent control group would be the 
most appropriate evaluation design to determine the students' 
progress toward the objectives. 

The carpentry teachers did not discover that they were using the 
new lathe improperly until the middle of the final semester. 

It appeared that the students who had attended the six out-of-class 
field sessions fared better on objective nine than did the remainder 
of the students. 

Students displayed a marked increase in career awareness as a 
result of the program. 
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Part Ikl: 
Group and Classroom Activities 



1 0 0 



PART III 



GROUP AND CLASSROOM ACTIVITIES 
Classroom Activities 

NOTE: The following activities are designed for use in the classroom to 
stimulate discussion on specific topics covered in this module. The 
activities are designed to be used following student self-study; however, 
depending on the background and abilities of students, these activities 
ray not require previous study. All classroom activities are keyed to 
the Content Outline to indicate an appropriate point at which they might 
be presented. 



On the chalkboard, draw Figure 2 as it is shown in the Study 
Guide on p. 20. While discussing it with the class, draw arrows 
from one phase to the next showing their interrelationship. Use 
solid lines to indicate progression and dotted lines to indicate 
recycling. For example, the activity "Select Goals" would have a 
solid arrow leading to the phase Program Planning, indicating that 
once the goals were selected, the next logical step would be to 
begin planning the program. Each of the activities would have dotted 
arrows leading back into the phase during which it was performed and 
then back to the preceding activity. These dotted arrows indicate 
a recycling process that might (or must be) conducted in the event 
that the results of an activity required that part of the program 
be revised. 
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2. Si tuation (9) 

The program planners for the Vocational Education Curriculum Special- 
ist Project are almost ready to begin their activities. Before be- 
ginning, they want to be certain that the responsibility for important 
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jobs is assigned to appropriate persons. The people involved with 
the project fall into tnree general categories — instructional develop- 
ers, organizers, evaluators--and include teachers, writers, adminis- 
trators, and the evuluator. The class is to help the program planners 
assign the planning jobs described below to the appropriate individuals, 
that is, those immediately responsible for them. You may wish to read 
the job descriptions to the class for reproduce them) and then discuss 
why the people indicated were chosen. 



Job Descri ptions 

1. Recommending where to go or whom to consult 
for information that might be needed in se- 
lecting the program or aspects of the program. 
For example, providing names of vocational ed- 
ucation agencies that have lists of available 
programs . 

2. Suggesting procedures for collecting infor- 
mation about the program's progress at dif- 
ferent stages of its development. For example, 
determining whether written performance tests 
are necessary or appropriate and whether other 
measures will also be appropriate. 

3. Choosing a program that meets the needs of 
vocational education as identified in the needs 
assessment phase. For example, choosing one 
type of program over another. 

4. Identifying where trie program might be bud- 
geting too tightly for materials, thereby 
diminishing the quality of instruction. For 
example, demonstrating that the use of pro- 
grammed learning materials rather than written 
texts is likely to increase achievement. 

5. Participating in program-planning activities. 
Fo^ example, attending some or all planning 
meeti ngs . 



A ppropriate P erson(s) 
1. (Evaluator) 



2. (Evaluator-- The 
monitoring of pro- 
gram progress 
should be built 
into the program.) 

3. (Decision-maker) 



4. (Media special ists-- 
This is an imme- 
diate responsibility 
of members of the 
instructional staff 
(writers) . ) 

5. (All staff persons) 
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Job Descriptions (cont 'd. ) 

Placing restrictions on the field test of 
the program. For example, randomly assign- 
ing students to nrnnr*PK. 

Providing checks nn th* accuracy and rele- 
vance of the pronnWs academic subject 
matter. For example, checking to see that 
students are learnino the most recent in- 
formation on vocational legislative mandates. 

Identifying and preparing the important com- 
ponents of a proor^' : ^lan so that it incor- 
porates a description of the cost of the 
total evaluation. For example, detailing 
the evaluation budget, including the costs 
of administering and scoring measures in the 
field test operation. 

Suggesting methods or techniques for making 
teaching and learning as effective as pos- 
sible. For example, making provisions for 
individualizing instruction each time a new 
concept is to be learned. 

Estimating the degree to which people in- 
volved in the program operation are commit- 
ted to its systematic planning. For example, 
establishing the extent to which individuals 
believe that systematic program planning will 
result in better or improved learning 
materials . 



Appropriate P erson(s ) (cont 'd. 

6. (Evaluator--the evel- 
uator has a better grasp 
of research designs . ) 

7. (Subject matter experts) 



8. (Evaluator) 



/ CllK^r* 1 r^,-»4-4-<^w» <^ v/ r-v <^ v- 4- <~ 
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and media specialists.) 



10. (Evaluator) 



Si tuition 

Instructor Ames has had several years of teachino experience in a 
large electronics technology associate degree program at a local com- 
munity college. In talking to the employers of many of the program's 
graduates, he finds that the employers are not satisfied with che 
graduates 1 troubleshooting performance. Since the electronics tech- 
nology program is heavily loaded with classroom and laboratory 
troubleshooting experiences with a focus on both principles and ap- 
plications, Mr. Ames becomes rather annoyed, not to mention confused. 
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The other instructors in the electronics technology program are 
well qualified and seem to have done a good job in otherwise pre- 
paring their students. 

Mr. Ames and the other electronics instructors put their heads 
together and decide that their program lacks a high quality, easily 
programmed, troubleshooting simulator. There are three simulators 
on the market, one by the Do-All company for $7,000; one by the 
Do-Some company for $4,300; and one by tr Do-Little company for 
$1,500. The four instructors don't know wr.ich of the simulators 
would be best; the Dean of Vocational Education wonders if the 
problem is lack of simulator or lack of good instruction. 

There are two problems then: 1) which of the simulators is best 
for the situation (the instructors' problem), and 2) is the problem 
lack of good instruction or lack of a simulator (the dean's problem)? 
You are an evaluation consultant called in to assist the dean and 
the instructors with their proole-ms. Assume that all three simulators 
can be obtained on loan from the manufacturers for one year for 
field test purposes, and assume that twelve classes at the college 
emphasize troubleshooting. 

Have the students discuss what kind of evaluation design they would 
set up to determine: (1) if the program needs changing, (2) which 
simulator is best, and (3) if a simulator is needed at all. 

Answ er: A design that would allow random assignment of students 
to the program is desirable. The Pretest-Posttest Contrast Design 
would probacy accommodate the situation. Three of the treatments 
would be instruction with each of the simulators, one treatment 
would be just as it always has been (without a simulator), and one 
group might receive a treatment consisting of a new program without 
a simulator. See page 40 in this guide for an explanation of the 
Pretest-Posttest Contrast Design. fi 



-94- 



Discussion Questions 



A. Why must evaluators determine the context (or decision area) at each 
evaluation phase? 

(This activity serves about the same purpose as does selecting objec- 
tives before conducting a program. It guides the evaluators in per- 
forming the work that is needed for specific decisions.) 



B. What are the characteristics of each evaluation activity, and how 
are they related? 

(This question may. be answered most fully by outlining the CSE model 
on the board and discussing the activities of each phase Stress the 
differences in purpose.) 

C. What are some technical skills that an evaluator must possess or be 
familiar with in order to conduct a thorough, comprehensive evaluation? 

(Some necessary skills are: 

1. oral ard written communication; 

2. test a\\i measure construction and administration; 

3. ability to conduct or interpret statistical analysis.) 



D. What are some criteria that the evaluator must examine in the program 
planning phase? 

(some examples are: 

a. What demonstrated or identified needs serve as the basis for 
planning? 

b. Are costs considered? 

c. Is the proposed program feasible in terms of psychological or 
philosophical limits or constraints? 

d. Are provisions for continuous evaluation built into the program? 

e. Is the program a duplicate of an existing program? 

f. Who contributes to the planning process? 

g. What characteristics might enhance or limit a student's chances for 
success in the program? 

h. Are time lines established in the plan? 

i. How may evaluation data be recycled back into the plan? 

Many other questions can be asked regarding the efficacy of the proposed 
program; all of these should be aimed at ensuring efficient operation 
and needs satisfaction.) 
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What are some criteria that the evaluator might examine in the 
implementation evaluation phase? 

(Criteria include: 

1. Before instruction: Is everything that is retired to conduct the 
program ready to go? 

2. After instruction begins: Is the operation of the program 
successful when the plans are followed? 

3. At end of major unit: Do students or participants appear to be 
achieving the program objectives?) 



If a program has fivp major phases, how many times could implementation 
evaluation activities conceivably be conducted during the life of the 
program? 

(Twel ve--before and after instruction begins in each of the first four 
units, and at the end of each major unit. If the program is modifiable 
after the last unit, the number could go to fifteen, but usually the 
final evaluation is outcome.) 



What are some criteria that are examined in the progress evaluation 
phase? 

(Criteria incl ude : 

1. Are factors outside the program influencing results? 

2. Are students able to progress efficiently? 

3. Are budget limits being observed without impeding the proo-am? 

4. Exactly how much progress have the students made?) 



What makes the documentation phase different from the outcome phase? 

(The documentation phase is an impartial accounting of the actual 
operation of the program and a description of how it differed (if it 
did) from what was planned. In the outcome phase, data are collected 
and analyzed ; " order to determine the results of the program.) 



What < ■ -i some ecific types of data that one might collect for 
eva' : .inn a rew program in an occupation such as ( your choice )? 

(Am , i' 1 depend on the program chosen, but they should be of the 
type uiDle from specific measurable objectives.) 
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Part IV: 
Student Self -Check 
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PART IV 
STUDENT SELF-CHECK 



TO THE STUDENT: 

Unlike the other modules in this series, this student self-check involves 
a fairly lengthy writing task. It would be impossible to predict the 
form of your response if you are taking this examination as a pretest, 
but if you are taking it as a posttest, your answer should include those 
items and sections that have been indicated as common to evaluation 
reports. 

Read the following situation and then perform the tasks that are indi- 
cated. You may, in many instances, have to provide fictitious data and 
fictitious results, but be sure your answers include the necessary 
material and are written in a logical, readable form. 

SITUATION : 

The United States Office of Education has released a Request for Proposals 
(RFP) to develop a program to train vocational education curriculum 
specialists. The RFP asks that the project be of two years duration and 
that all materials developed be in c form that may be used in under- 
graduate or graduate level university courses. Th* materials must also 
be in such a form that students can use th°m with a minimum of outside 
instruction. 

You are invited by the school of education of a university that is 
bidding on the project to serve as the external evaluator. Your first 
activity is to assist the program planners in preparing the proposal for 
the project. The proposal must contain the goals and objectives of the 
project, an outline or description of activities to be performed, mate- 
rials to be generated, a time schedule giving the dates at which portions 
of the project will start and end. and a budget for the project. 
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Assume then that the project is granted to the university ind is funded 
for a total of $1 million for the two years, and that you are retained 
as the project evaluator. 

TASKS : 

1. Develop an evaluation plan for a decision-facilitation evaluation 
of the project. (This would normally be included with the project 
proposal . ) 

2. Select evaluation criteria that the dec; is ion-makers might need 
examined. Some criteria are necessarily relator] to ttie project 
objectives, so may tiave to write <j few objectives. Include 
other criteria, such as costs, instructional ef f ect i veness , etc., 
that require examination in order to facilitate doc i s ion-mak i ri<| 
by the project operators. 

'J. Delineate possible methods lor collecting dor. is ion information for 
this project, that is, what tochn iques or devices may be used to 
examine the criteria selected above. 

4. Construct a mock evaluation report for 1 each phase of the evaluation, 
[tie report* should be cons'. t.i-nt with a decision-facilitation 
evaluation u n utation, arid should contain all inf (jrni.it Ion required 
to make decisions concerning development, modification, continuation, 
or term i na t ion of ttie pro jec t . 
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PART V 
APPENDICES 

Appendix A: 

Possible Study Activity Responses 
GOAL 15.1 



Stage 1 
Purpose 
Deci sion 

Stage 2 
Purpose 
Decision 

Stage 3 
Purpose 
Deci sion 



Preformati ve 
find out what is 
goal and program 



needed and 
selection 



what program f i ts best 



Formati ve 

provide information about how the program is operating 
modify or change program 

Summati ve 

provide information on what happenel 
continue or discontinue 



2. Activities (Phases) 



(1) 
(2) 
(3) 

(4) 
(5) 
(6) 

Processes 

(D 

(?.) 

(3) 
(4) 



Needs assessment 
Program planning 
Implementation evaluation 
Progress evaluation 
Documentation evaluation 
Outcome evaluation 



determine the decision area of concern 
select appropriate information 
collect and analyze data 
report summary information 



tval uator Acti vit ies 

(There are a multitude of activities that could be listed here. Try to 
justify your answers as best as possible based on the readings. After 
you have finished reading the material for the next two objectives, 
return and rocheck your answers.) 
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3. (This exercise could be completed in iany ways. Check your answer to 
ensure that you have stated the purpose as it is in the reading. Each 
phase provides the basis upon which the next phase is conducted, i.e., 
the results of the needs assessment phase aid the decision-maker and 
the evaluator in conducting the program planning activities, etc.) 



GOAL 15.2 



1 . (a) needs assessment 

(b) outcome 

(c) program planning 

(d) needs assessment 

(e) progress 

(f) program planning 

(g) implementation 

(h) documentation 



(i) implementation 

(j) outcome 

(k) needs assessment 

(1) progress 

(m) outcome 

(n) implementation 

(o) progress 

(p) program planning 



GOAL 15.3 



(1 

(2; 

(3) 
(4; 
(5i 

(6) 
(7; 



f 

d 

a 
d 

c 
e 
b 
b 



(9) 
(10) 
(11) 
(12) 
(13) 
(14) 
(15) 



f 
a 
e 
f 
d 
c 
b 



2. 



(1 

(2; 
(3; 
(4; 
(5] 



c 
a 
c 
b 
a 



(6) 
(7) 
(8) 
(9) 
(10) 



c 
b 
c 
a 
a 



(1 

(2; 
(3; 
(4; 
(5; 

(e; 
(7; 
(s; 
(9; 
(10; 



f 

d 

c 
a 
b 
f 
e 
e 
d 
a 



(11) b 

(12) e 

(13) c 

(14) b 

(15) b 

(16) c 

(17) c or d 

(18) f 

(19) b 

(20) d 
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4. 



(1) 






(16) 






(2) 






(17) 






(3) 


No: 


belongs in program planning 


(18) 






(4) 






(19) 


M — 

No : 


implementation 


(5) 


No: 


belongs in program planning 


(20) 






(6) 






(21) 


No: 


outcome 


(7) 






(22) 






(8) 






(23) 


No: 


progress 


(9) 


No: 


goes m needs assessment 


(24) 






(10) 


No : 


either prngress or outcome 


(25) 






(11) 


No: 


needs assessment 


(25) 






(12) 






(27) 






(13) 


No: 


Oli •--.L.V 


(28) 


No: 


documentation 


(14) 






(29) 






(15) 






(30) 







5. a. a. hypothesis testing procedures 
b. estimation procedures 

b. Educational researchers are typically concerned with the existence 
of a relationship tpcv.<*en two variables— they want to be able to 
generalize; evaluai.: ' want to know how much the relationship is, 
or what the magnitude of differences is— they are interested in 
particulars . 



GOAL 15.4 



1. (a) 


program planning 


(k) 


progress 


(b) 


implementation 


(1) 


program planning 


(c) 


needs assessment 


(m) 


documentation 


(d) 


outcome 


(n) 


outcome or progress 


(e) 


program planning 


(o) 


outcome 


(f) 


progress 






(g) 


needs assessment 






(h) 


progress or outcome 






(i) 


implementation 






(j) 


documentation 
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Appendix B: 

Possible Self-Check Responses 



Procedures for Conducting Evaluations of Vocational Education 

To the Instructor : Unlike the other modules in this series, this student 
self-check scoring key does not provide specific answers to the tasks 
the students are required to complete. It does provide a list of items 
that could be included, but because so much of the answer might be a re- 
flection of the students' various writing techniques, you are requested 
to use considerable flexibility when scoring it. 

Task 1 - Develop an Evaluat i on Plan 
(Items that could be included: 

a. the evaluator's role in determining the need for a program; 

b. the evaluator's role in setting goals and generating objectives; 

c. the evaluator's role in examining and defining the type of 
program needed to meet the delineated objectives; 

d. the evaluator's role in the selection of a program and plan 
for meeting the objectives; 

e. a plan for evaluating the effectiveness or impact of the proposed 
materials ; 

f. a plan for determining adequacy of curriculum content; 

g. a plan for determining the effectiveness of the processes required 
(or used) to implement the program; and 

h. a plan for determining the combined impact of the process and 
product on the students and on vocational education. 

The students might also include a flow chart indicating major project and 
evaluation activities and their interrelationships during the two-year 
life of the project.) 
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I 

Task 2 - Select Evaluation Criteria 

(The types of evaluation criteria students select could be judged by 
how well (and if) they answer the following questions. Students might 
even ask similar questions instead of listing specific criteria. Both 
criteria and questions, however, should pertain to the evaluation phase 
for which they are relevant. 



Needs Assessment 



a. Is there a need for a program as outlined in the Request 
* for Proposal s? 

b. What competencies are needed (or possessed) by people who 
are vocational curriculum specialists? 

c. Will, or should, the program be useful to other areas of 
vocational education or education in general? 

d. What evidence is there that once trained, the program parti- 
cipants will actually be involved in vocational curriculum 
development? 

e. Is there a logical progression from the general competence 
base to the stated goals and objectives or do objectives 
have a strong relationship to competence statements? 

f. Are the modules' objectives and learning activities logically 
related? 

g. Do the stated goals and objectives represent a comprehensive 
program? 

h. What is the value of curriculum goals, objectives, and the 
content areas addressed to vocational education professional 
development? 



Program Planning 

a. How may the program be useful to students of varying educa- 
tional (and vocational) backgrounds? 

b. Instead of creating a whole new program, could students take 
other courses in general education and relate the experiences 
and competencies gained to vocational education? 

c. Given all constraints, what type of materials should be de- 
veloped to meet the objectives? 

d. What evidence is there that once developed (in any form) the 
materials will be adopted and used by schools or institutions? 

115 
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e. Is an advisory committee used? Who are the members? 

f. Does the planned program relate to the specked needs? 

g. Will f he proposed program be feasible in terms of facilities 
and costs required to implement it? 

h. Can the curriculum be used as a part of existing courses? 

i. Is the curriculum useful as^a new course in vocational educa- 
tion preparation programs? 

j. Does the installation of the program require resources that 
are not readily available at the adopting institution? 

k. Is there consistency as to curriculum organization, sequencing, 
and levels of difficulty? 

1. Are the competencies that are specified as intended outcomes 
for the trainees consistent with the goals and needs of voca- 
tional education and of the user training institution or 
training group? 

m. Are there a variety of methods for attaining each training 
goal ? 

n. Do the competencies upon which the curriculum is built relate 
directly to the desired roles of a vocational education cur- 
riculum special ist? 

o. Does the variety of learning activities reflect several of the 
service areas and occupations included in vocational education? 

p. Is the competency base of the curriculum well defined and 
clearly delineated? 

q. Was the procedure used to establish the initial competency 
base adequate? 

r. Was the initial statement of competency adequately revised 
based on information gained from surveys , ,1 i terature review, 
advisory panel input, pilot test, etc.? 

s. Are supportive services adequate? 

t. Is continuous evaluation built into the program plan? 

u. Are people capable of teaching the program available? Are 
they needed? 

v. Does the proposed program contain provisions for modification 
during implementation? 

w. Does the program facilitate a genuine confrontation with the 
real it, <)f vocational education? 
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Implementation Evaluation 

a. Have all materials been completed on time? 

b. Have all persons who will field test the materials been 
properly and completely oriented? 

c. Are all planned procedures being followed? 

d. What procedures not planned are being followed? 

e. What outside (or non-program) factors may be contributing 
to the success or failure of the students? 

f. Are the curriculum materials ready for use or do they require 
additional effort? 

g. Do curriculum resources suit instructor preferences? 

h. Is the program (or are the materials) ready for use in the 
form in which it is being disseminated? 

i. What is the interest level of student materials? 

All questions or criteria in the implementation phase should relate 
to the implementation of the program as it was planned. 



P rogress Evaluation 

a. How much progress has been made toward specified objectives? 

b. Are expenses being kept within specified limits? 

c. Are students ready to progress to the next unit of instruction? 
Could they have progressed sooner? 

d. Does the training program function properly at the institution 
or with the group implementing it? 

e. Does the curriculum adequately cover: 

(1) basic concepts in vocational education? 

(2) the issues of (1) vocational education and the learner, 
(2) design, (3) development, (4) implementation and ^ 
management, and (5) evaluation of vocational education 
programs? 

(3) the issues of (1) curriculum management in contemporary 
vocational education and (2) professional leadership 
training? 

(4) field work experience in (1) project design and adminis- 
tration, (2) operation of school programs, (3) evaluation 
of school programs, (4) educational research and develop- 
ment, and (5) program supervision? 

1 1 7 
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f. Do individuals participating in the program acquire the 
knowledge, skills, and attitudes specified as the intended 
outcomes or is there an increase in the student's knowledge, 
skills, and attitudes in a positive direction as a result of 
the training program? 

g. Is the extent of the increase in line with the desired or 
projected increase? 

h. Is there a change in student gain attributable to differences 
in (a) program applications, (b) curriculum sequencing, (c) 
delivery mode, (d) resource application, or (e) other factors? 

i. Do different groups of students (classified by institution of 
enrollment, mode of study, background, etc.) show significantly 
different outcomes? 

j. Do different program sequences lead to differences in student 
outcomes? 

k. Is there evidence for the cumulative effect of exposure to 
more than a single aspect of the program (such as a unit of 
instruction)? 

1. Is there evidence for the effect of practice on material 
proficiency tests? 

m. What do students perceive they have learned from the materials? 

n. Does the teaching staff prefer to use this curriculum as opposed 
to that previously used? 

o. Does the program facilitate a genuine confrontation with the 
reality of vocational education? 

Questions or criteria in this section should relate to what is happening 
or resulting as a consequence of the program or materials. 



Documentation Evaluation 

a. Does the use of the program result in added or reduced implemen- 
tation costs? 

b. Is the program flexible enough to allow installation in a 
variety of institutions in the form of (a) a full degree program, 
(b) part of an ongoing program, (c) independent study, (d) in- 
service training, or (e) external and extension offering? 

c. Does the program allow for credit by examination? 

d. Were all planned procedures followed? 

e. What deviations from the planned program occurred? Why? 
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All questions or criteria in this section should be aimed at describing 
and documenting exactly what happened in the program. 



Outcome Evaluation 

a. What evidence is there that concerns the cost-effectiveness 
of the program? 

b. How are the effectiveness, adaptability, and feasibility 
viewed by those who are/were involved in the installation, 
management, implementation, and evaluation of the program? 

c. Do program participants exhibit more confidence in carrying 
out their responsibilities in vocational education? 

d. Is the program useful in the training of vocational education 
professionals in general? 

e. Is the program useful in the training of vocational education 
curriculum specialists? 

f. What evidence is there that the program participants are 
effective in planning, designing, implementing, and evaluating 
vocational education curriculum? 

g. Are employers of program participants satisfied with the 
(improved) performance of the participants? 

h. Is there any evidence that employers of potential program 
participants encourage their employees to enroll in the pro- 
gram? 

i. Is there any evidence that employers are interested in intro- 
ducing the program as an in-service program? 

All questions or criteria in this section should relate to the results 
or outcomes of the program. 



Task 3 - Data C ollect ion 

(The methods used to collect decision information (or data) will depend 
on the types of criteria that are selected for examination. The student 
should indicate in this section the evaluation design to be used, and 
the methods or instruments to be used to collect information within 
that design. Some examples of methods or techniques that the student 
could identify include: 
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a. observations (of students, program operators, and others); 

b. checklists (process measures) ; and 

c. criterion referenced measures (of students' abilities). 

Keep in mind that the measuring device or technique will depend entirely 
on the criteria selected for evaluation (or needed for decision-making). 

The evaluation ikuiyn should be of an experimental type (see pp. 63-65 
of the Study Guide) whenever possible in order to enhance the validity 
of the conclusions reached.) 



Task 4 - Evaluation Report 

(This portion of the student's answer should treat all six phases of 
decision-facilitation evaluation. Each phase report should contain the 
five sections specified by Hawkridge, Campeau, and Trickett (15): 
(a) a summary, (b) a description of the context in which the evaluation 
took place, (c) an explanation of the program, (d) a report of evaluation 
results, and (e) evaluator recommendations. You may wish to have the 
students write only one or two interim (or phase) evaluation reports 
and then outline a final report.) 
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